{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "利用LightGBM/XGboost实现Happy Customer Bank目标客户（贷款成功的客户）识别\n",
    "\n",
    "\n",
    "一、\t任务说明：Happy Customer Bank目标客户识别\n",
    "https://discuss.analyticsvidhya.com/t/hackathon-3-x-predict-customer-worth-for-happy-customer-bank/3802\n",
    "\n",
    "1)\t文件说明\n",
    "Train.csv：训练数据\n",
    "Test.csv：测试数据\n",
    "\n",
    "2)\t字段说明\n",
    "数据集共26个字段: 其中1-24列为输入特征，25-26列为输出特征。\n",
    "1.\tID - 唯一ID（不能用于预测）\n",
    "2.\tGender - 性别\n",
    "3.\tCity - 城市\n",
    "4.\tMonthly_Income - 月收入（以卢比为单位）\n",
    "5.\tDOB - 出生日期\n",
    "6.\tLead_Creation_Date - 潜在（贷款）创建日期\n",
    "7.\tLoan_Amount_Applied - 贷款申请请求金额（印度卢比，INR）\n",
    "8.\tLoan_Tenure_Applied - 贷款申请期限（单位为年）\n",
    "9.\tExisting_EMI -现有贷款的EMI（EMI：电子货币机构许可证） \n",
    "10.\tEmployer_Name雇主名称\n",
    "11.\tSalary_Account - 薪资帐户银行\n",
    "12.\tMobile_Verified - 是否移动验证（Y / N）\n",
    "13.\tVAR5 - 连续型变量\n",
    "14.\tVAR1-  类别型变量\n",
    "15.\tLoan_Amount_Submitted - 提交的贷款金额（在看到资格后修改和选择）\n",
    "16.\tLoan_Tenure_Submitted - 提交的贷款期限（单位为年，在看到资格后修改和选择）\n",
    "17.\tInterest_Rate - 提交贷款金额的利率\n",
    "18.\tProcessing_Fee - 提交贷款的处理费（INR）\n",
    "19.\tEMI_Loan_Submitted -提交的EMI贷款金额（INR）\n",
    "20.\tFilled_Form - 后期报价后是否已填写申请表格\n",
    "21.\tDevice_Type - 进行申请的设备（浏览器/移动设备）\n",
    "22.\tVar2 - 类别型变量\n",
    "23.\tSource - 类别型变量\n",
    "24.\tVar4 - 类别型变量\n",
    "\n",
    "输出：\n",
    "25.\tLoggedIn - 是否login（只用于理解问题的变量，不能用于预测，测试集中没有）\n",
    "26. Disbursed - 是否发放贷款（目标变量），1为发放贷款（目标客户）\n",
    "\n",
    "二、作业要求：\n",
    "1.\t适当的特征工程（20分）\n",
    "2.\t用LightGBM完成任务，并用交叉验证对模型的超参数（learning_rate、n_estimators、num_leaves、max_depth、min_data_in_leaf、colsample_bytree、subsample）进行调优。（70分）\n",
    "或者用XGBoost完成任务，并用交叉验证对模型的超参数（learning_rate、n_estimators、max_depth、min_child_weight、colsample_bytree、subsample、reg_lambda、reg_）进行调优。\n",
    "3.\t对最终模型给出特征重要性（10分）\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "#导入必要的工具包\n",
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>City</th>\n",
       "      <th>Monthly_Income</th>\n",
       "      <th>Loan_Amount_Applied</th>\n",
       "      <th>Loan_Tenure_Applied</th>\n",
       "      <th>Existing_EMI</th>\n",
       "      <th>Employer_Name</th>\n",
       "      <th>Salary_Account</th>\n",
       "      <th>Mobile_Verified</th>\n",
       "      <th>Var5</th>\n",
       "      <th>...</th>\n",
       "      <th>Interest_Rate</th>\n",
       "      <th>Processing_Fee</th>\n",
       "      <th>EMI_Loan_Submitted</th>\n",
       "      <th>Filled_Form</th>\n",
       "      <th>Device_Type</th>\n",
       "      <th>Var2</th>\n",
       "      <th>Source</th>\n",
       "      <th>Var4</th>\n",
       "      <th>LoggedIn</th>\n",
       "      <th>Disbursed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Female</td>\n",
       "      <td>Delhi</td>\n",
       "      <td>20000</td>\n",
       "      <td>300000.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>CYBOSOL</td>\n",
       "      <td>HDFC Bank</td>\n",
       "      <td>N</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>Web-browser</td>\n",
       "      <td>G</td>\n",
       "      <td>S122</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>Male</td>\n",
       "      <td>Mumbai</td>\n",
       "      <td>35000</td>\n",
       "      <td>200000.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TATA CONSULTANCY SERVICES LTD (TCS)</td>\n",
       "      <td>ICICI Bank</td>\n",
       "      <td>Y</td>\n",
       "      <td>13</td>\n",
       "      <td>...</td>\n",
       "      <td>13.25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6762.9</td>\n",
       "      <td>N</td>\n",
       "      <td>Web-browser</td>\n",
       "      <td>G</td>\n",
       "      <td>S122</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Male</td>\n",
       "      <td>Panchkula</td>\n",
       "      <td>22500</td>\n",
       "      <td>600000.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>ALCHEMIST HOSPITALS LTD</td>\n",
       "      <td>State Bank of India</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>Web-browser</td>\n",
       "      <td>B</td>\n",
       "      <td>S143</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Male</td>\n",
       "      <td>Saharsa</td>\n",
       "      <td>35000</td>\n",
       "      <td>1000000.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>BIHAR GOVERNMENT</td>\n",
       "      <td>State Bank of India</td>\n",
       "      <td>Y</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>Web-browser</td>\n",
       "      <td>B</td>\n",
       "      <td>S143</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>Male</td>\n",
       "      <td>Bengaluru</td>\n",
       "      <td>100000</td>\n",
       "      <td>500000.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>25000.0</td>\n",
       "      <td>GLOBAL EDGE SOFTWARE</td>\n",
       "      <td>HDFC Bank</td>\n",
       "      <td>Y</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>N</td>\n",
       "      <td>Web-browser</td>\n",
       "      <td>B</td>\n",
       "      <td>S134</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Gender       City  Monthly_Income  Loan_Amount_Applied  \\\n",
       "0  Female      Delhi           20000             300000.0   \n",
       "1    Male     Mumbai           35000             200000.0   \n",
       "2    Male  Panchkula           22500             600000.0   \n",
       "3    Male    Saharsa           35000            1000000.0   \n",
       "4    Male  Bengaluru          100000             500000.0   \n",
       "\n",
       "   Loan_Tenure_Applied  Existing_EMI                        Employer_Name  \\\n",
       "0                  5.0           0.0                              CYBOSOL   \n",
       "1                  2.0           0.0  TATA CONSULTANCY SERVICES LTD (TCS)   \n",
       "2                  4.0           0.0              ALCHEMIST HOSPITALS LTD   \n",
       "3                  5.0           0.0                     BIHAR GOVERNMENT   \n",
       "4                  2.0       25000.0                 GLOBAL EDGE SOFTWARE   \n",
       "\n",
       "        Salary_Account Mobile_Verified  Var5  ... Interest_Rate  \\\n",
       "0            HDFC Bank               N     0  ...           NaN   \n",
       "1           ICICI Bank               Y    13  ...         13.25   \n",
       "2  State Bank of India               Y     0  ...           NaN   \n",
       "3  State Bank of India               Y    10  ...           NaN   \n",
       "4            HDFC Bank               Y    17  ...           NaN   \n",
       "\n",
       "   Processing_Fee  EMI_Loan_Submitted  Filled_Form  Device_Type  Var2 Source  \\\n",
       "0             NaN                 NaN            N  Web-browser     G   S122   \n",
       "1             NaN              6762.9            N  Web-browser     G   S122   \n",
       "2             NaN                 NaN            N  Web-browser     B   S143   \n",
       "3             NaN                 NaN            N  Web-browser     B   S143   \n",
       "4             NaN                 NaN            N  Web-browser     B   S134   \n",
       "\n",
       "  Var4 LoggedIn Disbursed  \n",
       "0    1        0         0  \n",
       "1    3        0         0  \n",
       "2    1        0         0  \n",
       "3    3        0         0  \n",
       "4    3        1         0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#导入数据集\n",
    "train_data = pd.read_csv(\"./data/Train.csv\",encoding='ISO-8859-1').drop([\"ID\",\"Lead_Creation_Date\",\"DOB\"], axis=1)\n",
    "test_data= pd.read_csv(\"./data/Test.csv\",encoding='ISO-8859-1').drop([\"ID\",\"Lead_Creation_Date\",\"DOB\"], axis=1)\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 87020 entries, 0 to 87019\n",
      "Data columns (total 23 columns):\n",
      "Gender                   87020 non-null object\n",
      "City                     86017 non-null object\n",
      "Monthly_Income           87020 non-null int64\n",
      "Loan_Amount_Applied      86949 non-null float64\n",
      "Loan_Tenure_Applied      86949 non-null float64\n",
      "Existing_EMI             86949 non-null float64\n",
      "Employer_Name            86949 non-null object\n",
      "Salary_Account           75256 non-null object\n",
      "Mobile_Verified          87020 non-null object\n",
      "Var5                     87020 non-null int64\n",
      "Var1                     87020 non-null object\n",
      "Loan_Amount_Submitted    52407 non-null float64\n",
      "Loan_Tenure_Submitted    52407 non-null float64\n",
      "Interest_Rate            27726 non-null float64\n",
      "Processing_Fee           27420 non-null float64\n",
      "EMI_Loan_Submitted       27726 non-null float64\n",
      "Filled_Form              87020 non-null object\n",
      "Device_Type              87020 non-null object\n",
      "Var2                     87020 non-null object\n",
      "Source                   87020 non-null object\n",
      "Var4                     87020 non-null int64\n",
      "LoggedIn                 87020 non-null int64\n",
      "Disbursed                87020 non-null int64\n",
      "dtypes: float64(8), int64(5), object(10)\n",
      "memory usage: 15.3+ MB\n"
     ]
    }
   ],
   "source": [
    "#看起来有很多类别行的属性需要做特征值处理\n",
    "train_data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Gender                       0\n",
       "City                      1003\n",
       "Monthly_Income               0\n",
       "Loan_Amount_Applied         71\n",
       "Loan_Tenure_Applied         71\n",
       "Existing_EMI                71\n",
       "Employer_Name               71\n",
       "Salary_Account           11764\n",
       "Mobile_Verified              0\n",
       "Var5                         0\n",
       "Var1                         0\n",
       "Loan_Amount_Submitted    34613\n",
       "Loan_Tenure_Submitted    34613\n",
       "Interest_Rate            59294\n",
       "Processing_Fee           59600\n",
       "EMI_Loan_Submitted       59294\n",
       "Filled_Form                  0\n",
       "Device_Type                  0\n",
       "Var2                         0\n",
       "Source                       0\n",
       "Var4                         0\n",
       "LoggedIn                     0\n",
       "Disbursed                    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### 查看是否有空值\n",
    "train_data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Gender                       0\n",
       "City                       398\n",
       "Monthly_Income               0\n",
       "Loan_Amount_Applied         40\n",
       "Loan_Tenure_Applied         40\n",
       "Existing_EMI                40\n",
       "Employer_Name               42\n",
       "Salary_Account            5037\n",
       "Mobile_Verified              0\n",
       "Var5                         0\n",
       "Var1                         0\n",
       "Loan_Amount_Submitted    14922\n",
       "Loan_Tenure_Submitted    14922\n",
       "Interest_Rate            25607\n",
       "Processing_Fee           25746\n",
       "EMI_Loan_Submitted       25607\n",
       "Filled_Form                  0\n",
       "Device_Type                  0\n",
       "Var2                         0\n",
       "Source                       0\n",
       "Var4                         0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Monthly_Income</th>\n",
       "      <th>Loan_Amount_Applied</th>\n",
       "      <th>Loan_Tenure_Applied</th>\n",
       "      <th>Existing_EMI</th>\n",
       "      <th>Var5</th>\n",
       "      <th>Loan_Amount_Submitted</th>\n",
       "      <th>Loan_Tenure_Submitted</th>\n",
       "      <th>Interest_Rate</th>\n",
       "      <th>Processing_Fee</th>\n",
       "      <th>EMI_Loan_Submitted</th>\n",
       "      <th>Var4</th>\n",
       "      <th>LoggedIn</th>\n",
       "      <th>Disbursed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>count</td>\n",
       "      <td>8.702000e+04</td>\n",
       "      <td>8.694900e+04</td>\n",
       "      <td>86949.000000</td>\n",
       "      <td>8.694900e+04</td>\n",
       "      <td>87020.000000</td>\n",
       "      <td>5.240700e+04</td>\n",
       "      <td>52407.000000</td>\n",
       "      <td>27726.000000</td>\n",
       "      <td>27420.000000</td>\n",
       "      <td>27726.000000</td>\n",
       "      <td>87020.000000</td>\n",
       "      <td>87020.000000</td>\n",
       "      <td>87020.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>mean</td>\n",
       "      <td>5.884997e+04</td>\n",
       "      <td>2.302507e+05</td>\n",
       "      <td>2.131399</td>\n",
       "      <td>3.696228e+03</td>\n",
       "      <td>4.961503</td>\n",
       "      <td>3.950106e+05</td>\n",
       "      <td>3.891369</td>\n",
       "      <td>19.197474</td>\n",
       "      <td>5131.150839</td>\n",
       "      <td>10999.528377</td>\n",
       "      <td>2.949805</td>\n",
       "      <td>0.029350</td>\n",
       "      <td>0.014629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>std</td>\n",
       "      <td>2.177511e+06</td>\n",
       "      <td>3.542068e+05</td>\n",
       "      <td>2.014193</td>\n",
       "      <td>3.981021e+04</td>\n",
       "      <td>5.670385</td>\n",
       "      <td>3.082481e+05</td>\n",
       "      <td>1.165359</td>\n",
       "      <td>5.834213</td>\n",
       "      <td>4725.837644</td>\n",
       "      <td>7512.323050</td>\n",
       "      <td>1.697720</td>\n",
       "      <td>0.168785</td>\n",
       "      <td>0.120062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>min</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.000000e+04</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>11.990000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>1176.410000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25%</td>\n",
       "      <td>1.650000e+04</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000e+05</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>15.250000</td>\n",
       "      <td>2000.000000</td>\n",
       "      <td>6491.600000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50%</td>\n",
       "      <td>2.500000e+04</td>\n",
       "      <td>1.000000e+05</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.000000e+05</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>4000.000000</td>\n",
       "      <td>9392.970000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>75%</td>\n",
       "      <td>4.000000e+04</td>\n",
       "      <td>3.000000e+05</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.500000e+03</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>5.000000e+05</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>6250.000000</td>\n",
       "      <td>12919.040000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>max</td>\n",
       "      <td>4.445544e+08</td>\n",
       "      <td>1.000000e+07</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>1.000000e+07</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>3.000000e+06</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>37.000000</td>\n",
       "      <td>50000.000000</td>\n",
       "      <td>144748.280000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Monthly_Income  Loan_Amount_Applied  Loan_Tenure_Applied  Existing_EMI  \\\n",
       "count    8.702000e+04         8.694900e+04         86949.000000  8.694900e+04   \n",
       "mean     5.884997e+04         2.302507e+05             2.131399  3.696228e+03   \n",
       "std      2.177511e+06         3.542068e+05             2.014193  3.981021e+04   \n",
       "min      0.000000e+00         0.000000e+00             0.000000  0.000000e+00   \n",
       "25%      1.650000e+04         0.000000e+00             0.000000  0.000000e+00   \n",
       "50%      2.500000e+04         1.000000e+05             2.000000  0.000000e+00   \n",
       "75%      4.000000e+04         3.000000e+05             4.000000  3.500000e+03   \n",
       "max      4.445544e+08         1.000000e+07            10.000000  1.000000e+07   \n",
       "\n",
       "               Var5  Loan_Amount_Submitted  Loan_Tenure_Submitted  \\\n",
       "count  87020.000000           5.240700e+04           52407.000000   \n",
       "mean       4.961503           3.950106e+05               3.891369   \n",
       "std        5.670385           3.082481e+05               1.165359   \n",
       "min        0.000000           5.000000e+04               1.000000   \n",
       "25%        0.000000           2.000000e+05               3.000000   \n",
       "50%        2.000000           3.000000e+05               4.000000   \n",
       "75%       11.000000           5.000000e+05               5.000000   \n",
       "max       18.000000           3.000000e+06               6.000000   \n",
       "\n",
       "       Interest_Rate  Processing_Fee  EMI_Loan_Submitted          Var4  \\\n",
       "count   27726.000000    27420.000000        27726.000000  87020.000000   \n",
       "mean       19.197474     5131.150839        10999.528377      2.949805   \n",
       "std         5.834213     4725.837644         7512.323050      1.697720   \n",
       "min        11.990000      200.000000         1176.410000      0.000000   \n",
       "25%        15.250000     2000.000000         6491.600000      1.000000   \n",
       "50%        18.000000     4000.000000         9392.970000      3.000000   \n",
       "75%        20.000000     6250.000000        12919.040000      5.000000   \n",
       "max        37.000000    50000.000000       144748.280000      7.000000   \n",
       "\n",
       "           LoggedIn     Disbursed  \n",
       "count  87020.000000  87020.000000  \n",
       "mean       0.029350      0.014629  \n",
       "std        0.168785      0.120062  \n",
       "min        0.000000      0.000000  \n",
       "25%        0.000000      0.000000  \n",
       "50%        0.000000      0.000000  \n",
       "75%        0.000000      0.000000  \n",
       "max        1.000000      1.000000  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#对数据值型特征，用常用统计量观察其分布\n",
    "train_data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Gender属性的不同取值和出现的次数\n",
      "Male      49848\n",
      "Female    37172\n",
      "Name: Gender, dtype: int64\n",
      "\n",
      "Gender属性的不同取值和出现的次数\n",
      "Male      49848\n",
      "Female    37172\n",
      "Name: Gender, dtype: int64\n",
      "\n",
      "City属性的不同取值和出现的次数\n",
      "Delhi        12527\n",
      "Bengaluru    10824\n",
      "Mumbai       10795\n",
      "Hyderabad     7272\n",
      "Chennai       6916\n",
      "             ...  \n",
      "UDWADA           1\n",
      "Bandipore        1\n",
      "CHOTILA          1\n",
      "Dantewada        1\n",
      "Beawar           1\n",
      "Name: City, Length: 697, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#可以看到空值很多，需要对空值做处理\n",
    "categorical_features = ['Gender','Gender','City']\n",
    "for col in categorical_features:\n",
    "    print('\\n%s属性的不同取值和出现的次数'%col)\n",
    "    print(train_data[col].value_counts())\n",
    "    #train_data[col] = train_data[col].astype('object')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#数据清洗以及编码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "#去掉重复的数据\n",
    "train_data = train_data.drop_duplicates()\n",
    "test_data=test_data.drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Gender                   0\n",
       "City                     0\n",
       "Monthly_Income           0\n",
       "Loan_Amount_Applied      0\n",
       "Loan_Tenure_Applied      0\n",
       "Existing_EMI             0\n",
       "Employer_Name            0\n",
       "Salary_Account           1\n",
       "Mobile_Verified          0\n",
       "Var5                     0\n",
       "Var1                     0\n",
       "Loan_Amount_Submitted    0\n",
       "Loan_Tenure_Submitted    0\n",
       "Interest_Rate            0\n",
       "Processing_Fee           0\n",
       "EMI_Loan_Submitted       0\n",
       "Filled_Form              0\n",
       "Device_Type              0\n",
       "Var2                     0\n",
       "Source                   0\n",
       "Var4                     0\n",
       "LoggedIn                 0\n",
       "Disbursed                0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data=train_data.fillna(method = 'backfill', axis = 0)\n",
    "#train_data.interpolate(method = 'linear', axis = 0)\n",
    "test_data=test_data.fillna(method = 'backfill', axis = 0)\n",
    "train_data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#目前看来有一条Salary_Account的记录没有得到清除，由于这个数据样本比较小，可以删除此列。\n",
    "train_data=train_data.dropna(axis = 0)\n",
    "test_data=test_data.dropna(axis = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.isnull().sum().sum()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get_dummies()可以直接对字符型变量进行one-hot编码，对于特征值变量做特征编码，日期处理比较麻烦由于时间问题，暂时不处理会丢弃\n",
    "for col in train_data.columns :\n",
    "    if train_data[col].dtype=='object':\n",
    "        train_data[col] = pd.get_dummies(train_data[col])\n",
    "        train_data[col]\n",
    "train_data.head()\n",
    "\n",
    "for col in test_data.columns :\n",
    "    if test_data[col].dtype=='object':\n",
    "        test_data[col] = pd.get_dummies(test_data[col])\n",
    "        test_data[col]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型建立，基于xgboost来建立模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "#测试数据和训练数据进一步分离"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features = ['Disbursed']\n",
    "y_train=train_data[categorical_features]\n",
    "X_train=train_data.drop(['LoggedIn','Disbursed'], axis=1)\n",
    "X_train_array = np.array(X_train)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "def modelfit(alg, X_train_value, y_train_value, cv_folds=None, early_stopping_rounds=10):\n",
    "    xgb_param = alg.get_xgb_params()    \n",
    "    #直接调用xgboost，而非sklarn的wrapper类\n",
    "    xgtrain = xgb.DMatrix(X_train_value, label = y_train_value)\n",
    "    cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], folds =cv_folds,\n",
    "             metrics='logloss', early_stopping_rounds=early_stopping_rounds)\n",
    "  \n",
    "    cvresult.to_csv('1_nestimators.csv', index_label = 'n_estimators')\n",
    "    \n",
    "    #最佳参数n_estimators\n",
    "    n_estimators = cvresult.shape[0]\n",
    "    \n",
    "    # 采用交叉验证得到的最佳参数n_estimators，训练模型\n",
    "    alg.set_params(n_estimators = n_estimators)\n",
    "    alg.fit(X_train_value, y_train_value, eval_metric='logloss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\miniconda_anzhuang\\lib\\site-packages\\sklearn\\preprocessing\\label.py:219: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "d:\\miniconda_anzhuang\\lib\\site-packages\\sklearn\\preprocessing\\label.py:252: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "xgb1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=1000,  #数值大没关系，cv会自动返回合适的n_estimators,时间关系 n_estimators 以及depth都比较小\n",
    "        max_depth=3,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel=0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n",
    "modelfit(xgb1, X_train_array, y_train, cv_folds = kfold)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "#n_estimators调优"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEXCAYAAACgUUN5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXQd9Xn/8fdz79ViS5YtW5KxLWN5N4aAAWNMFiABGiCJIYQQSMgvkDRp+iuHpElLoM3Jr6WhbdKsLZwkJAHSJkAI2QhxlrJmZbHZbWMsjMHyKm/aLEu69z6/P2ZkrsWVLRuNRtJ8XufM8cx3vnfmmRl5njvfmTtfc3dERCS5UnEHICIi8VIiEBFJOCUCEZGEUyIQEUk4JQIRkYRTIhARSTglApECZvYPZvaduOMQGUpKBCOMmVWa2QYze39B2Tgze8XMLi4oW2xm95rZbjPbY2arzewGM6sO519hZjkzaw+H9Wb21xHHfqaZNUW5jsNRLB53/1d3/8uI1rfBzM6OYtlRGKrjNdL2y2ikRDDCuHs78DHg62ZWGxZ/EVjh7ncDmNkbgYeAPwIL3H0CcC6QBU4oWNyf3b3S3SuBi4EvmtmJQ7MlcjjMLBN3DDKKubuGETgAtwF3AGcCO4EpBfP+APzXIT5/BfCHPmWPAe8vmF4GrAL2ECSWYwrmHROW7QnrLCuYdz6wGmgDNgF/B1QAnUAeaA+Hqf1s103AL8PPPwrMHsD+WAD8L7ALWAtcciTxAP8EfD/8XAPgwJXARmA38HHgFOCZcNtvLFjPbOCB8HjsAH4ATAjn/U+4rs5wXdcMYB9vAD4TrqsLyITTm8JtWQucVWRfLAW2AumCsncDz4TjS4AVQCuwDfhKP/v0TKCpn3njgf8GmoGXgc8CqXBeGvhyuA9eAq4K92Omn2VtAM7uZ95HgcbwuN7T+zcDGPBVYDvQEu6j4/o73nH/fx3uQ+wBaDjCAwfVwJbwP9uVBeUVQA448xCfv4KCRBCe3PYA88LpeUAHcA5QAlwT/ocsDacbgX8Ip98W/qebH352C/CWgjhPCsf7PbEUxHFb+J9+SXji+wFw5yE+U0Fwor4y/MxJ4X459nDjoXgi+CZQDvwFsA/4GVAHTAtPRGeE9eeE+6sMqAV+B3ytYNkHnPAOto8L6j8FTAfGAPPD7ZxaEF/RJAm8CJxTMP0j4Npw/M/AB8PxSmBpP8vo93gRJIGfA+PCOF4APhLO+zjBibg+3N/3cQSJIPy72hEezzLgv4DfhfPeDqwEJhAkhWMIvwz1d7w19D+oaWiEcvfdBN8kxwI/KZhVTdDkt7W3wMy+GN4n6DCzzxbUXRqWtxNcDfwPsC6c9z7gl+7+v+7eA3yJ4GT0RoJvnJXAv7t7t7s/ANwLXBZ+tgdYaGZV7r7b3Z84zM37ibs/5u5ZgkSw6BD13wlscPdb3T0bru/HBM1dgxHPv7j7Pnf/LcGJ+w533+7um4DfAycCuHtjuL+63L0Z+ApwxkGWe7B93Os/3X2ju3cSJPiycFtK3H2Du7/Yz7LvIDweZjaO4FvyHQX7Y46Z1bh7u7s/cjg7w8zSYezXuXubu28guAL4YFjlEuDr7t4U/p3+++Esv8AHgFvc/Ql37wKuA04zs4ZwG8YRXAmau69x9y0F2/d6jnfiKBGMUGZ2OcE3sfuALxTM2k3QBDGlt8Ddr/HgPsFPCb4x93rE3Sd4cI/gKOBY4F/DeVMJLvl7l5En+DY6LZy3MSzr9XI4D+A9BCeel83sYTM77TA3b2vB+F6CpHMwM4BTw6S2x8z2EJxEjhqkeLYVjHcWma4EMLM6M7vTzDaZWSvwfaDmIMs92D7utbFgfiPwSYKrlu3huqb2s+zbgYvMrAy4CHjC3XvX9RGCq5HnzexxM3vnQWIspobgSvDlgrLC4z+1MO4+44ej7/5pJ2h2mxZ++biRoBlxm5ndbGZVYdXXe7wTR4lgBDKzOoL20Y8CfwVcYmanA7h7B0G7+kWHs0x330bwLfpdYdFmghNs7zqNoIliUzhvupkV/v0cHc7D3R939wsImk9+BtzVu5rDiekwbAQeDpNa71Dp7n89xPH8W7jM4929CricoNmiV9/1HWwfF/2Mu9/u7m8OP+cc+CWgsN5qgpPoecD7CRJD77x17n4Zwf74AnC3mVUMfDPZQfCte0ZB2f7jT9A0U18wb/phLLtQ3/1TAUzi1b+z/3T3kwm+wMwD/j4s7+94Sz+UCEamG4GfufuD4eXwNcC3w29/hNMfNrNrw6SBmdUDM/tboJlNIrihuCosugt4h5mdZWYlwKcJblj+iSDRdADXmFmJmZ1JkEDuNLNSM/uAmY0PmztaCZo0IPgmPcnMxg/Sfuh1LzDPzD4YxlNiZqeY2TFDHM84ghvBe8xsGuGJqcA2YFbB9MH28WuY2Xwze1t4nPcRXI3kitUN3Q5cDZxOcI+gdzmXm1lteAWyJyzudzlmVl44EFxx3gXcED66PAP4FMEVUO92fcLMppnZBIIb3IdS0mc9mTD+K81sUbjN/wo86u4bwuN7arjfOsL9kTvE8Zb+xH2TQsPhDcCFBN+UJvQpvx+4oWD6VGA5wX/0PcBzwA3ApHD+FQT/QXqfmNlO0IZcV7CMdxPc9GsBHia8+RrOOzYsawnrvDssLwV+TdBE1Qo8Dry54HO3EFze76H/p4Y+XzB9Joe4wRzWm0/wpFFzuPwHCO4tHFY8FL9ZnCmo30TBjXiCk99nC/bJynB/PkVwYm8qqHsB8Eq4rr8bwD7ewIE3l48nuJfTRnBD/d5i+7Cg/tEEJ+1f9in/fni82wkS/4X9fP7McPv7DnMI7kV9P9zfG4HP8epTQxmCK9adBE8N/S3BFYT1s54NRdbx+XDexwlufPdub31YfhbBk0LtvPqEVuWhjreG4oOFO1VEJBJmdh7wTXefccjKEgs1DYnIoDKzMWZ2vpllwiay/0fwoIIMU7oikBHBzN4C/KrYPA+eepJhwszGEjRzLSC4j/FL4BPu3hprYNIvJQIRkYRT05CISMKNuBdZ1dTUeENDQ9xhiIiMKCtXrtzh7rXF5o24RNDQ0MCKFSviDkNEZEQxs5f7m6emIRGRhFMiEBFJOCUCEZGEUyIQEUk4JQIRkYRTIhARSbhIE4GZnWtma82s0cyu7afOJWa22sxWmdntxeqIiEh0IksEYXd2NxF0jLEQuMzMFvapM5eg+7k3ufuxBL0vRWL96sd45K7/wPP5Q1cWEUmQKK8IlgCN7r7e3buBOwnex17oo8BNHvRrirtvjyqY7U8sZ+nqz9PasiuqVYiIjEhRJoJpHNhXaRMH9sUKQfdy88zsj2b2iJmdG1UwJROD3vJ2be6vr28RkWSKMhFYkbK+rzrNAHMJekK6DPhO2LXdgQsy+5iZrTCzFc3NzUcUTEVt0CdG67ZXjujzIiKjVZSJoIkDO62uJ+hisW+dn7t7j7u/BKwlSAwHcPeb3X2xuy+urS36zqRDqp4SdNe7b6cSgYhIoSgTwePAXDObaWalwKXAPX3q/Ax4K4CZ1RA0Fa2PIpiao44m50a+pSmKxYuIjFiRJQJ3zwJXAb8B1gB3ufsqM7vezJaF1X4D7DSz1cCDwN+7+84o4klnSthhE8m09b0oERFJtkhfQ+3uy4Hlfco+VzDuwKfCIXK7M3WM2bd1KFYlIjJiJOqXxR3lkxnfHdkTqiIiI1KiEkFPxRRq8836UZmISIFEJQLG11NuPezZpasCEZFeiUoEZRPrAdi1OZIHk0RERqREJYKKugYA2rb323WniEjiJCoRTDwq+FFZl35UJiKyX7ISweR6uj2tH5WJiBRIVCJIpdPsSE2ipF0/KhMR6ZWoRACwJ1PH2H3b4g5DRGTYSFwi6BwzmfE9enxURKRX4hJBd8UUavM7yedycYciIjIsJC4RpMbXU2pZdjXrPoGICCQwEZSGPZXt3vJSzJGIiAwPiUsE4yY3APpRmYhIr8QlgolhT2Xdu/SjMhERSGAiqK6ZQpeXQMumuEMRERkWEpcILJWiOTWJko4tcYciIjIsJC4RALSU1DFWPZWJiAAJTQR7xxxFdU9z3GGIiAwLiUwE2cqp1PhOctls3KGIiMQukYkgNeFoMpanebN+SyAikshEMHbyHAB2bnwh5khEROKXyEQwafp8ADq2NcYciYhI/BKZCOrqZ9HjaXI71XexiEgiE0GmpJRtqVpKWvXrYhGRRCYCgN2lU6nqVJeVIiKJTQR7K4+mLqtXUYuIJDYR+IQZTKCdlt074g5FRCRWkSYCMzvXzNaaWaOZXVtk/hVm1mxmT4XDX0YZT6GyutkANL/y/FCtUkRkWIosEZhZGrgJOA9YCFxmZguLVP2huy8Kh+9EFU9fVVPmAtC6ed1QrVJEZFiK8opgCdDo7uvdvRu4E7ggwvUdlroZCwDoatYjpCKSbFEmgmnAxoLpprCsr/eY2TNmdreZTS+2IDP7mJmtMLMVzc2D87K4ceMnspsqUns2DMryRERGqigTgRUp8z7TvwAa3P144D7ge8UW5O43u/tid19cW1s7aAFuz0yhokO/JRCRZIsyETQBhd/w64EDntd0953u3hVOfhs4OcJ4XqN9bD0Tu9VBjYgkW5SJ4HFgrpnNNLNS4FLgnsIKZjalYHIZsCbCeF6ju2oGdflmerq7Dl1ZRGSUiiwRuHsWuAr4DcEJ/i53X2Vm15vZsrDa1Wa2ysyeBq4GrogqnmIyk2aSsTzbNurlcyKSXJkoF+7uy4Hlfco+VzB+HXBdlDEcTMVRc+Fp2N20lvrZx8YVhohIrBL7y2KAmv2vo34x5khEROKT7EQwZQZdXgK71FOZiCRXohNBKp1ma3oypW16hFREkivRiQBgT9k0xu/T66hFJLkSnwj2VU5ncnYrns/HHYqISCwSnwi8dQuV1snO7ZviDkVEJBaJTwQVp30EgG3rn405EhGReCQ+EdTNOh6AtqZVMUciIhIPJYL62ez1MmheG3coIiKxSHwisFSKTSXTqWjVj8pEJJkSnwgAWsbOpK7r5bjDEBGJhRIBkJ04j8nspL11d9yhiIgMOSUCoHTKMQBsbnwm5khERIaeEgEwqeENALS88lzMkYiIDD0lAmDqzGPo8TTZ7c/HHYqIyJBTIgBKSsvYnJ5K+R51UCMiyaNEENo5poGazg1xhyEiMuSUCELdE2YzJb+V7q59cYciIjKklAhCmckLyFiezet1w1hEkkWJIDTh6OMA2LVBL58TkWRRIghNnRO8fK5rq54cEpFkUSIIja0cz1ZqKdm1Lu5QRESGlBJBge3lM6jeq47sRSRZlAgK7K2azdRsE/lcLu5QRESGjBJBgVTdAsZYN5s36D6BiCSHEkGB6tknA7B93YqYIxERGTpKBAWmzz+ZnBtdTU/HHYqIyJBRIihQPraSjel6xuxcHXcoIiJDJtJEYGbnmtlaM2s0s2sPUu9iM3MzWxxlPAOxo3I+UzpfiDsMEZEhE1kiMLM0cBNwHrAQuMzMFhapNw64Gng0qlgOR7b2WCazk93NW+IORURkSER5RbAEaHT39e7eDdwJXFCk3r8AXwSGxdveKmYEN4w3Pf9YzJGIiAyNKBPBNGBjwXRTWLafmZ0ITHf3ew+2IDP7mJmtMLMVzc3Ngx9pgWkLgtap9pefjHQ9IiLDRZSJwIqU+f6ZZingq8CnD7Ugd7/Z3Re7++La2tpBDPG1JtZNYzsTyWzXW0hFJBmiTARNwPSC6Xpgc8H0OOA44CEz2wAsBe4ZDjeMt4yZy6R23TAWkWSIMhE8Dsw1s5lmVgpcCtzTO9PdW9y9xt0b3L0BeARY5u6x/5qrc9JCpuc2sq+zI+5QREQid8hEYGazzawsHD/TzK42swmH+py7Z4GrgN8Aa4C73H2VmV1vZsteb+BRKpt2AhnLs3HtE3GHIiISuYFcEfwYyJnZHOC7wEzg9oEs3N2Xu/s8d5/t7jeEZZ9z93uK1D1zOFwNANTNOwWA3S+ujDkSEZHoDSQR5MNv9+8GvubufwtMiTaseE1pOIZ2H0N+yzNxhyIiErmBJIIeM7sM+BDQ+5hnSXQhxS+VTrOxdBbjW/QWUhEZ/QaSCK4ETgNucPeXzGwm8P1ow4pf6/gFTO9er74JRGTUO2QicPfV7n61u99hZtXAOHf/9yGILVapqSdQaZ1sbFRn9iIyug3kqaGHzKzKzCYCTwO3mtlXog8tXnXHvAmAbWv+GHMkIiLRGkjT0Hh3bwUuAm5195OBs6MNK37T551Im4/BN+qdQyIyug0kEWTMbApwCa/eLB71Uuk0L5cvYNIeNQ2JyOg2kERwPcGPwl5098fNbBawLtqwhof22hNpyL7E3vaWuEMREYnMQG4W/8jdj3f3vw6n17v7e6IPLX5jZy0lY3leevZPcYciIhKZgdwsrjezn5rZdjPbZmY/NrP6oQgubke/4XQAWtcpEYjI6DWQpqFbCV4WN5WgP4FfhGWj3oTaKTTZFMq26p1DIjJ6DSQR1Lr7re6eDYfbgGg7BRhGto47jqP3Pofn83GHIiISiYEkgh1mdrmZpcPhcmBn1IENF7lpi6lhD1s3NsYdiohIJAaSCD5M8OjoVmALcDHBaycSYdL84Idlm577fcyRiIhEYyBPDb3i7svcvdbd69z9QoIflyXCjIVL2OclZF9+NO5QREQicaQ9lH1qUKMYxkpKy3ipdB4Tdj0ddygiIpE40kRQrGP6Uatl0iJm9TSyr3Nv3KGIiAy6I00EPqhRDHNls95IqWVZ/7TuE4jI6NNvIjCzNjNrLTK0EfymIDFmn/wX5N1oWf1A3KGIiAy6TH8z3H3cUAYynFVNquPFzEzGbf1z3KGIiAy6I20aSpzmmiXM7VrNvs6OuEMRERlUSgQDNGbuGZRZD41PPhR3KCIig0qJYIBmLv4Lcm606T6BiIwySgQDVDWhhpdKZjN+m35YJiKjy0BeQ13s6aGN4aupZw1FkMPFjtpTmdO9hs6O9rhDEREZNAO5IvgK8PcEr6CuB/4O+DZwJ3BLdKENP2PmnUmpZWl84v64QxERGTQDSQTnuvu33L3N3Vvd/WbgfHf/IVAdcXzDyuyTzyHrKdqefzDuUEREBs1AEkHezC4xs1Q4XFIw76C/MDazc81srZk1mtm1ReZ/3MyeNbOnzOwPZrbwcDdgKFVWVfNiyVyqt+s+gYiMHgNJBB8APghsD4cPApeb2Rjgqv4+ZGZp4CbgPGAhcFmRE/3t7v4Gd18EfJGgGWpY2113KnO619Lepg7tRWR0GMhrqNe7+7vcvSYc3uXuje7e6e5/OMhHlwCN4ee7Ce4pXNBn2a0FkxWMgHcYjVt4NiWW44VHlscdiojIoIiy8/ppwMaC6aawrO/y/8bMXiS4Irh6oIHHZe4pb6fDy+le8+u4QxERGRRRdl5f7FXVr/nG7+43ufts4DPAZ4suyOxjZrbCzFY0NzcPYNXRKS0r54XKxTTs+oP6MRaRUSHKzuubgOkF0/XA5oPUvxO4sNgMd7/Z3Re7++La2oGsOlrZWWdzFDtYv3pF3KGIiLxuUXZe/zgw18xmmlkpcCnBlcV+Zja3YPIdwLqBBh6nmae9G4BtK+85RE0RkeGv39dQF/gwcCPwVYKmnT8xgM7r3T1rZlcBvwHSwC3uvsrMrgdWuPs9wFVmdjbQA+wGPnRkmzG0aqY28GJ6FhOa9HsCERn5DpkI3P0VYFlhmZl9EvjaAD67HFjep+xzBeOfGHCkw0zzlDNYvPF77Nm5nQmT6uIOR0TkiKnz+iNUfcI7yViedX9W85CIjGzqvP4IzTnxTHYzDl74TdyhiIi8Luq8/gilMxlerFrKnNY/k8vl4g5HROSIqfP618HmvZ1q2nj+sf+NOxQRkSPWbyJw93HuXlVkGOfuA3naaNRbcPrFdHkJbU/cHXcoIiJHTD2UvQ4VVdWsqVzCrOb71TwkIiOWEsHrlDvmQurYpeYhERmxlAhepwWnv1fNQyIyoikRvE4VVdWsrjxVzUMiMmIpEQyCvJqHRGQEUyIYBAtOv5h9XkLbEz+KOxQRkcOmRDAIgqeHTmVW8wNqHhKREUeJYJD4sRdRxy5W/emXcYciInJYlAgGycIz30crFXQ99r24QxEROSxKBIOkfGwlz9e8nTe0PkzLrh1xhyMiMmBKBINo4ps/TLn1sOa+W+IORURkwJQIBtGcE97M+nQDE9feFXcoIiIDpkQwmMzYMecS5uXW0fjco3FHIyIyIEoEg2z+OR+m29M0/+67cYciIjIgSgSDbHzNFJ4b92YWbF/Ovs69cYcjInJISgQRKD3lCqpp45nf/nfcoYiIHJISQQSOfcsFvJKaxoRnvoPn83GHIyJyUEoEEbBUmq0LrmRebh2rH7sv7nBERA5KiSAibzj/r2ilgs7f3xh3KCIiB6VEEJExlVWsmXIRJ7b/js0b1sYdjohIv5QIItRw/idwjA2/+nrcoYiI9EuJIEKTp8/l2arTOW7rz2hr3R13OCIiRSkRRGzcWz9BlXXw7M++GncoIiJFRZoIzOxcM1trZo1mdm2R+Z8ys9Vm9oyZ3W9mM6KMJw5zTnobz5WfxIL1t9LR1hJ3OCIirxFZIjCzNHATcB6wELjMzBb2qfYksNjdjwfuBr4YVTxxKj3rH5hIK0/rqkBEhqEorwiWAI3uvt7du4E7gQsKK7j7g+7e+x6GR4D6COOJzbxTzmFV2YnMf/EWOtpb4w5HROQAUSaCacDGgummsKw/HwF+VWyGmX3MzFaY2Yrm5uZBDHHoZM66jkm06KpARIadKBOBFSnzohXNLgcWA/9RbL673+zui919cW1t7SCGOHTmL3k7q8tOYF6jrgpEZHiJMhE0AdMLpuuBzX0rmdnZwD8Cy9y9K8J4Ypc+6x+pYQ9P/fDzcYciIrJflIngcWCumc00s1LgUuCewgpmdiLwLYIksD3CWIaF+UvezlOVb2HRK7extemluMMREQEiTATungWuAn4DrAHucvdVZna9mS0Lq/0HUAn8yMyeMrN7+lncqDH5PV+ghCwv/+i6uEMREQEgE+XC3X05sLxP2ecKxs+Ocv3D0ZSZx/LY1PdxyuY7WPvk75l/4lviDklEEk6/LI7Bwks/T4uNo2f5deqvQERip0QQg8rxk1jJfI7reZZH7/1O3OGISMIpEcTkbZ/9Nesy85j7xOfZ2bw17nBEJMGUCGKSymQoffeNjPc21n3/k3GHIyIJpkQQoxnHnsoT9R9kacuvePLhn8cdjogklBJBzE64/F9pSk2h9qFraNcvjkUkBkoEMSsbU0nHOV+m3rfyzHevijscEUkgJYJhYP5p72DF1Mt54+6f89jy78UdjogkjBLBMLHoii/TmJnL/EevY9OGdXGHIyIJokQwTGRKy6l4/22UWJbdP7iCnp6euEMSkYRQIhhGpsw6jhdO/ieO63mOR759ddzhiEhCKBEMM4ve9X9ZWXcRb9l+O3/8yU1xhyMiCaBEMAwt+ui3WFN2Aouf/n88+9gDcYcjIqOcEsEwlC4ppf6v7mJXqpqjln+Yppd181hEoqNEMEyNm3gUuffdQTn7yN52Ic3bXtO5m4jIoFAiGMbqFyxm2/m3MSW/jZ03X0DL7l1xhyQio5ASwTA3Z8m5vHjmjczJNrLhGxexd29H3CGJyCijRDACLHzrpaxe/HlO6H6SF772TlrbWuIOSURGESWCEeL4d/0NT590A8d3PcnLX38He9RMJCKDRIlgBDlh2VWsOu3LHNOzii03nkvzti1xhyQio4ASwQjzhnM/wgtn3Mjs7It0fvOtNK55Ou6QRGSEUyIYgRa+7QM0LbuTKm+n5s7zeeJ3v4g7JBEZwZQIRqhZJ59Dz5X30ZaZwHH3f4iHv38D+Vw+7rBEZARSIhjBamcsYNLVD7Ou8hTOaPwij33pQnbt3BF3WCIywigRjHBjx9ew8NPLeXLeJ1m89/e0/9ebefLP98UdloiMIEoEo4Cl0pz4/n+madldlFs3x//6Yh686Sra2tvjDk1ERgAlglGk4eRzqPrUSp6rfSdvbf4fmr+8lMd+9yvcPe7QRGQYUyIYZcrHVXPCVd+n8ZxbqaSTJQ9cyh+/9F5efmVD3KGJyDAVaSIws3PNbK2ZNZrZtUXmn25mT5hZ1swujjKWpJnzpouovuZJnprxYU5tf4CJ313K/TdfQ/Mu/SJZRA4UWSIwszRwE3AesBC4zMwW9qn2CnAFcHtUcSRZyZgqFl35VVo//Hs2Vp3EWZu/BV9fxH23/Qu7W1rjDk9EhokorwiWAI3uvt7du4E7gQsKK7j7Bnd/BtAD8BGaNONYFn56OZsu+hl7xjZw9oYvkf3Kcdx382do2rI17vBEJGZRJoJpwMaC6aaw7LCZ2cfMbIWZrWhubh6U4JJo2vFvZe41D7PxXT9kV+U8zt78TcZ/cxEPfPUKVjz+J/J53VQWSaJMhMu2ImVHdKZx95uBmwEWL16ss9XrYcb0k8+Fk8+led1jbPv1l3jzzl9Q+suf8tSvjqV5zntZeNblTJtcG3ekIjJEokwETcD0gul6QP0tDiO1c5dQO/cuulq2seq332LK87ez6IV/omPtv/Hw2LeQW/hujnvLMuomVMUdqohEyKJ6xtzMMsALwFnAJuBx4P3uvqpI3duAe9397kMtd/Hixb5ixYpBjlYAcGfbcw+y44/fY8bW31LJXlp9LE+OWUrXrL+g4dR3MPfo6ZgVu9gTkeHMzFa6++Ki86L8sZGZnQ98DUgDt7j7DWZ2PbDC3e8xs1OAnwLVwD5gq7sfe7BlKhEMDe/pZNMTv6Zl5Y+Z3vwQVd5G1lM8l5rP1prTKJtzBnNOOpPptRPiDlVEBiC2RBAFJYIY5LLsfOHPbF35c8ZtfIj6rkZSOJ1eyurUPLZPWERqxqlMXvBG5s1qYGxplC2OInIklAhkUOU7drH5mftoXf0AldtXMrWrkUz4BPAmr2F9yVxaxx+DHXUs4xsWMX3mAqZNrCCdUpOSSFyUCCRaXe3sWvcIO154lPzmJ6nes5ra7GZS4UNie72Ml5jCttIZ7B3XQK56FuV1cxk/dTZ1k6cxtXos5SXpmDdCZHRTIpkWszIAAAs3SURBVJCh19VOx6bnaG5cSdeW1WR2rmN8x0vU5LYfUG2vl7HZJ9GcrqW1dDKdY6aQq5xCuuooSqunUTFxKlWTjqKmaizVFaVUlKZ1s1rkCBwsEagxV6JRVknFrKVUzFp6YHlPJ7ldG9i9cQ3t29bTs/NlrGUj9Xu3UtX1OOO7dsOeAz+Sc2MX49jkVbQwjo50FZ2Z8XSXTiBXNh4vr4YxE0iNrSY9ZjwlY4OhrKKKsWMrGVtWQkVZmjGlaSpKM5SXpNVMJVJAiUCGVskY0pOPoWbyMdQUm5/tgrat9LRsoa35FTp3b6G7ZRv51m2Udu7kqK49lHZvZ0x2HRUdrWQ6sgddXdZTdFBOO2No8XI2U85eL2OfldOTKqMnVUYuVU42VUYuHYznM+Xk02WQKcPTZZAuwzKlUFKOpcsgU4JlykhlSrF0KalMOJSUkc6UkSopIZUuIZ3OkE6nyaSMdMpe/TdtpCwYT5lhBumUkTYjFZaljODf3nIDKyw3A2N/udFbP/jXeLXcwjoi/VEikOElUwbVMyipnsHEhqUHr+sO3R2wbw907qanfTddHXvo6thDz949ZDvbyO9rI7+vFbo7KO3uoLynndpsJ6lsB5ncDkry+8jku8j0dFHa3U1qkF971eNpsgRDjhQ9pMn1TnuKHCly++elyJIiT1CeJU3eU+Sw/WW5gvk5Uji2v8wxcp4ij4XDq3U9LHNSYCmyDnmMVCpNTx4cI51K05UHJ0Umk2ZfNqhTWpJmX4+TJ0VZSZrOHiePMaY0Q0d3HgfKy0rY25UjT4qxZRnauoLyirJS2rpyOEZFeSkt+4LxyjHBeI4UE8aWsmtvFseorixnR3sPeYy6qnK2tvbgwOTxY9nSug8w6qrGsKW1C8c4qqr81fHxY9nUsg8H6qvHsnH3PvIY0yZW8MqusHxiBRt3dQIwfVIlL+/ciwMzJlXy0q695DFm1Ixj/Y6gvKGmCsd4cUcHALNrx9HY3I6TYnZdJeu2B5+ZO7mKtdvacYx5k6t4fls7BiyYMo7nt7QBcMyUKtZsaQULxnuTdC8r+jKG3kQezL/yTQ2cdczkQf0bBd0jEHmVO+R6INsJ2W7I7guuUHJdwb/ZLsh1Q66HfHYfuWw3uZ5u8j37yGV7yGe7yHV34bkePNdDPtcDuSyez+LhOPlgmlwPeA7L5yCfBc9DPgeexcJx82xQZ/90cHrfP+5BijDPg3vBdO+4gwfpAA/SQG+5eZBCzNmfImTw5YPT/f4kHOxlI29G3oMEbJYimw/KUykjuz8xG9m84xipdJpszrllzBV85tp/PqJYdI9AZCDMIFMaDIeQCoeSyIMaIu7hkAfPhf+GyYnCeR5MH1Cee7W8sM4By8y/Or/3s/lcP8ssst7C5RVbT7G6/cXQu71F482/dnv3f5aDrC9PsfhSYb0UBcvpG1vR7Sm+rs8cf04kh1+JQERebX8ghU4LyaOuKkVEEk6JQEQk4ZQIREQSTolARCThlAhERBJOiUBEJOGUCEREEk6JQEQk4UbcKybMrBl4+Qg/XgPsGMRwRoKkbbO2d3RL2vbC4G3zDHevLTZjxCWC18PMVvT3ro3RKmnbrO0d3ZK2vTA026ymIRGRhFMiEBFJuKQlgpvjDiAGSdtmbe/olrTthSHY5kTdIxARkddK2hWBiIj0oUQgIpJwiUkEZnauma01s0YzuzbueAabmU03swfNbI2ZrTKzT4TlE83sf81sXfhvddyxDiYzS5vZk2Z2bzg908weDbf3h2Z26O7GRhAzm2Bmd5vZ8+GxPm00H2Mz+9vw7/k5M7vDzMpH0zE2s1vMbLuZPVdQVvR4WuA/w3PYM2Z20mDFkYhEYGZp4CbgPGAhcJmZLYw3qkGXBT7t7scAS4G/CbfxWuB+d58L3B9OjyafANYUTH8B+Gq4vbuBj8QSVXS+Dvza3RcAJxBs+6g8xmY2DbgaWOzuxwFp4FJG1zG+DTi3T1l/x/M8YG44fAz4xmAFkYhEACwBGt19vbt3A3cCF8Qc06By9y3u/kQ43kZwgphGsJ3fC6t9D7gwnggHn5nVA+8AvhNOG/A24O6wymjb3irgdOC7AO7e7e57GMXHmKDfzDFmlgHGAlsYRcfY3X8H7OpT3N/xvAD4bw88AkwwsymDEUdSEsE0YGPBdFNYNiqZWQNwIvAoMNndt0CQLIC6+CIbdF8DrgF6exafBOxx92w4PdqO8yygGbg1bA77jplVMEqPsbtvAr4EvEKQAFqAlYzuYwz9H8/IzmNJSQRWpGxUPjdrZpXAj4FPuntr3PFExczeCWx395WFxUWqjqbjnAFOAr7h7icCHYySZqBiwrbxC4CZwFSggqB5pK/RdIwPJrK/76QkgiZgesF0PbA5plgiY2YlBEngB+7+k7B4W+/lY/jv9rjiG2RvApaZ2QaCpr63EVwhTAibEWD0HecmoMndHw2n7yZIDKP1GJ8NvOTuze7eA/wEeCOj+xhD/8czsvNYUhLB48Dc8GmDUoIbTvfEHNOgCtvHvwuscfevFMy6B/hQOP4h4OdDHVsU3P06d6939waC4/mAu38AeBC4OKw2arYXwN23AhvNbH5YdBawmlF6jAmahJaa2djw77t3e0ftMQ71dzzvAf5P+PTQUqCltwnpdXP3RAzA+cALwIvAP8YdTwTb92aCy8RngKfC4XyCdvP7gXXhvxPjjjWCbT8TuDccnwU8BjQCPwLK4o5vkLd1EbAiPM4/A6pH8zEG/hl4HngO+B+gbDQdY+AOgvsfPQTf+D/S3/EkaBq6KTyHPUvwNNWgxKFXTIiIJFxSmoZERKQfSgQiIgmnRCAiknBKBCIiCadEICKScEoEIiIJp0QgMgBmtsjMzi+YXjZYrzM3s0+a2djBWJbIkdDvCEQGwMyuIPgBz1URLHtDuOwdh/GZtLvnBjsWSSZdEcioYmYNYYct3w47NPmtmY3pp+5sM/u1ma00s9+b2YKw/L1hRyhPm9nvwteSXA+8z8yeMrP3mdkVZnZjWP82M/tG2DHQejM7I+xwZI2Z3Vawvm+Y2Yowrn8Oy64meKHag2b2YFh2mZk9G8bwhYLPt5vZ9Wb2KHCamf27ma0OOyn5UjR7VBIh7p9Ya9AwmAPQQNBJz6Jw+i7g8n7q3g/MDcdPJXhfEQQ/358Wjk8I/70CuLHgs/unCToXuZPgFQAXAK3AGwi+aK0siKX3VQFp4CHg+HB6A1ATjk8leMdOLcHbRh8ALgznOXBJ77KAtbx6VT8h7n2vYeQOuiKQ0egld38qHF9JkBwOEL6u+43Aj8zsKeBbQG8nH38EbjOzjxKctAfiF+7uBElkm7s/6+55YFXB+i8xsyeAJ4FjCXrL6+sU4CEP3riZBX5A0BkNQI7g7bIQJJt9wHfM7CJg7wDjFHmNzKGriIw4XQXjOaBY01CKoIOTRX1nuPvHzexUgt7PnjKz19Q5yDrzfdafBzJmNhP4O+AUd98dNhmVF1lOsXfO99rn4X0Bd8+a2RKCN3JeClxF8CpukcOmKwJJJA867XnJzN4L+zsGPyEcn+3uj7r754AdBO+AbwPGvY5VVhF0JNNiZpM5sIOVwmU/CpxhZjVhX9uXAQ/3XVh4RTPe3ZcDnyR4K6nIEdEVgSTZB4BvmNlngRKCdv6ngf8ws7kE387vD8teAa4Nm5H+7XBX5O5Pm9mTBE1F6wman3rdDPzKzLa4+1vN7DqCd+4bsNzdi71vfxzwczMrD+v97eHGJNJLj4+KiCScmoZERBJOTUMy6pnZTQR9HBf6urvfGkc8IsONmoZERBJOTUMiIgmnRCAiknBKBCIiCadEICKScP8fJor4R3hQQ9gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cvresult =  pd.read_csv('1_nestimators.csv')\n",
    "        \n",
    "# plot\n",
    "test_means = cvresult['test-logloss-mean']\n",
    "test_stds = cvresult['test-logloss-std'] \n",
    "        \n",
    "train_means = cvresult['train-logloss-mean']\n",
    "train_stds = cvresult['train-logloss-std'] \n",
    "\n",
    "x_axis = range(0, cvresult.shape[0])\n",
    "        \n",
    "pyplot.errorbar(x_axis, test_means, yerr=test_stds ,label='Test')\n",
    "pyplot.errorbar(x_axis, train_means, yerr=train_stds ,label='Train')\n",
    "pyplot.title(\"XGBoost n_estimators vs Log Loss\")\n",
    "pyplot.xlabel( 'n_estimators' )\n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'n_estimators4_1.png' )\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "#可以看到 当n_estimators=98的时候，模型的性能最佳。n_estimators 最小。learning_rate 就固定为0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\miniconda_anzhuang\\lib\\site-packages\\sklearn\\preprocessing\\label.py:219: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "d:\\miniconda_anzhuang\\lib\\site-packages\\sklearn\\preprocessing\\label.py:252: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=3, shuffle=True),\n",
       "             error_score='raise-deprecating',\n",
       "             estimator=XGBClassifier(base_score=0.5, booster='gbtree',\n",
       "                                     colsample_bylevel=0.7, colsample_bynode=1,\n",
       "                                     colsample_bytree=0.8, gamma=0,\n",
       "                                     learning_rate=0.1, max_delta_step=0,\n",
       "                                     max_depth=3, min_child_weight=1,\n",
       "                                     missing=None, n_estimators=102, n_jobs=1,\n",
       "                                     nthread=None, objective='binary:logistic',\n",
       "                                     random_state=0, reg_alpha=0, reg_lambda=1,\n",
       "                                     scale_pos_weight=1, seed=3, silent=None,\n",
       "                                     subsample=0.3, verbosity=1),\n",
       "             iid='warn', n_jobs=-1,\n",
       "             param_grid={'max_depth': range(3, 10, 2),\n",
       "                         'min_child_weight': range(1, 6, 2)},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#第一轮参数调整得到调整max_depth，min_child_weight\n",
    "\n",
    "xgb2 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=98,  #第一轮参数调整得到的n_estimators最优值  98\n",
    "        max_depth=5,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n",
    "max_depth = range(3,10,2)\n",
    "min_child_weight = range(1,6,2)\n",
    "param_test2 = dict(max_depth=max_depth, min_child_weight=min_child_weight)\n",
    "\n",
    "gsearch2 = GridSearchCV(xgb1, param_grid = param_test2, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch2.fit(X_train , y_train)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "({'max_depth': 3, 'min_child_weight': 1}, -0.06744169325884568)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " #最佳解\n",
    "gsearch2.best_params_,     gsearch2.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "#调整colsample_bytree & colsample_bylevel\n",
    "xgb3 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=98,  #第一轮参数调整得到的n_estimators最优值  98\n",
    "        max_depth=3,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\miniconda_anzhuang\\lib\\site-packages\\sklearn\\preprocessing\\label.py:219: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "d:\\miniconda_anzhuang\\lib\\site-packages\\sklearn\\preprocessing\\label.py:252: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "({'colsample_bytree': 0.9, 'subsample': 0.8}, -0.06726080432702941)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "subsample = [i/10.0 for i in range(3,9)]\n",
    "colsample_bytree = [i/10.0 for i in range(6,10)]\n",
    "param_test3= dict(subsample=subsample, colsample_bytree=colsample_bytree)\n",
    "\n",
    "gsearch3 = GridSearchCV(xgb3, param_grid = param_test3, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch3.fit(X_train , y_train)\n",
    "\n",
    "gsearch3.best_params_,     gsearch3.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([12.9856029 , 13.87473817, 14.61679435, 14.28976312, 14.00823236,\n",
       "        13.64291492, 13.28931136, 14.31725659, 14.6780509 , 14.78132806,\n",
       "        14.44740691, 14.09083319, 14.22184534, 15.12667718, 15.73429298,\n",
       "        15.96090164, 15.87323833, 15.43901649, 14.68804855, 15.76820455,\n",
       "        16.36443934, 16.44408002, 15.99555516, 15.20864034]),\n",
       " 'std_fit_time': array([0.21722117, 0.21337832, 0.23786214, 0.09492494, 0.21565471,\n",
       "        0.16213661, 0.27142558, 0.11429027, 0.08299799, 0.05006575,\n",
       "        0.07062455, 0.05137844, 0.0521931 , 0.14146365, 0.17077176,\n",
       "        0.11117906, 0.25067158, 0.10700637, 0.23096659, 0.22589528,\n",
       "        0.14149513, 0.1406716 , 0.06241992, 0.40821181]),\n",
       " 'mean_score_time': array([0.15173163, 0.16293998, 0.1492404 , 0.15161381, 0.1452702 ,\n",
       "        0.15772672, 0.15432568, 0.14953628, 0.15589328, 0.15282698,\n",
       "        0.13979917, 0.14876528, 0.14867587, 0.14736953, 0.15411654,\n",
       "        0.15159044, 0.15302958, 0.14338994, 0.16573472, 0.14237423,\n",
       "        0.14370751, 0.14100757, 0.11880522, 0.10691028]),\n",
       " 'std_score_time': array([0.00737565, 0.01800513, 0.01377772, 0.02287542, 0.00634516,\n",
       "        0.01518028, 0.01680939, 0.0070689 , 0.01703945, 0.01933952,\n",
       "        0.02254786, 0.01155209, 0.01384953, 0.01105494, 0.01432735,\n",
       "        0.01064564, 0.01121469, 0.00982801, 0.01442669, 0.00755537,\n",
       "        0.01709521, 0.01615267, 0.02005194, 0.00724103]),\n",
       " 'param_colsample_bytree': masked_array(data=[0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.7, 0.7, 0.7, 0.7, 0.7,\n",
       "                    0.7, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.9, 0.9, 0.9, 0.9,\n",
       "                    0.9, 0.9],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_subsample': masked_array(data=[0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.3, 0.4, 0.5, 0.6, 0.7,\n",
       "                    0.8, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.3, 0.4, 0.5, 0.6,\n",
       "                    0.7, 0.8],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'colsample_bytree': 0.6, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.6, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.7, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.8, 'subsample': 0.8},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.3},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.4},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.5},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.6},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.7},\n",
       "  {'colsample_bytree': 0.9, 'subsample': 0.8}],\n",
       " 'split0_test_score': array([-0.06929796, -0.06939547, -0.06934092, -0.06893632, -0.06893229,\n",
       "        -0.0690508 , -0.06930812, -0.06923929, -0.06946813, -0.06885701,\n",
       "        -0.06887577, -0.06900467, -0.06902253, -0.06919348, -0.06912747,\n",
       "        -0.06911441, -0.06903876, -0.06910724, -0.06928666, -0.06931093,\n",
       "        -0.06903798, -0.06904686, -0.06885637, -0.06888616]),\n",
       " 'split1_test_score': array([-0.06813679, -0.06780413, -0.06771318, -0.06774031, -0.06765354,\n",
       "        -0.06753806, -0.06815921, -0.06781582, -0.06761165, -0.06774466,\n",
       "        -0.06763858, -0.06750514, -0.06780772, -0.06784793, -0.06764444,\n",
       "        -0.06752529, -0.0677554 , -0.06752629, -0.06796892, -0.0677753 ,\n",
       "        -0.06756605, -0.06761198, -0.06755869, -0.06741659]),\n",
       " 'split2_test_score': array([-0.06769182, -0.06735425, -0.06748006, -0.06734779, -0.06738568,\n",
       "        -0.06730525, -0.06739373, -0.06715024, -0.06730022, -0.06717323,\n",
       "        -0.0670548 , -0.06704672, -0.06765051, -0.06721374, -0.0674206 ,\n",
       "        -0.0674386 , -0.06725725, -0.06708788, -0.06744457, -0.06758986,\n",
       "        -0.06727581, -0.06728288, -0.06716835, -0.06706905]),\n",
       " 'split3_test_score': array([-0.06559504, -0.06521529, -0.06510666, -0.06512234, -0.06527674,\n",
       "        -0.06521416, -0.0652833 , -0.06533721, -0.06544599, -0.0655602 ,\n",
       "        -0.06518633, -0.06520349, -0.06468101, -0.0648806 , -0.06477783,\n",
       "        -0.06503687, -0.06477483, -0.06484315, -0.06504701, -0.06497599,\n",
       "        -0.06522152, -0.06507241, -0.06493281, -0.06501492]),\n",
       " 'split4_test_score': array([-0.06809746, -0.0678996 , -0.06795422, -0.06797925, -0.0680542 ,\n",
       "        -0.06792044, -0.06801795, -0.06826035, -0.06790352, -0.06795653,\n",
       "        -0.06786544, -0.06806502, -0.06798171, -0.06790628, -0.06773694,\n",
       "        -0.06785995, -0.06783848, -0.06791681, -0.0680662 , -0.06778162,\n",
       "        -0.06819891, -0.0677006 , -0.06786023, -0.06791721]),\n",
       " 'mean_test_score': array([-0.06776384, -0.06753377, -0.06751903, -0.06742522, -0.06746051,\n",
       "        -0.06740576, -0.06763249, -0.0675606 , -0.06754592, -0.06745834,\n",
       "        -0.0673242 , -0.06736503, -0.06742872, -0.06740843, -0.06734148,\n",
       "        -0.06739505, -0.06733297, -0.0672963 , -0.0675627 , -0.06748677,\n",
       "        -0.06746007, -0.06734297, -0.06727531, -0.0672608 ]),\n",
       " 'std_test_score': array([0.00120935, 0.00134761, 0.00136874, 0.00126481, 0.0012108 ,\n",
       "        0.00124913, 0.00132735, 0.00130265, 0.00128818, 0.00109276,\n",
       "        0.00122001, 0.00126265, 0.00145524, 0.00141811, 0.00141537,\n",
       "        0.00132303, 0.00140665, 0.00139847, 0.00139521, 0.00140087,\n",
       "        0.00127215, 0.00128549, 0.00129775, 0.00127887]),\n",
       " 'rank_test_score': array([24, 19, 18, 12, 16, 10, 23, 21, 20, 14,  4,  8, 13, 11,  6,  9,  5,\n",
       "         3, 22, 17, 15,  7,  2,  1])}"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch3.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.067261 using {'colsample_bytree': 0.9, 'subsample': 0.8}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEGCAYAAACtqQjWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOydeVyUVRfHv5ddZF9lEUHFXdxzz4xcUNM0tVwqKyvrbbHF1rdsz1bttU2zstJKsVJz19zSzB0URQVBBUR22WSdue8fMxAgqMDADHC/n898ZuaZ+9znPFPOj3PuuecIKSUKhUKhUBgCM2MboFAoFIrGgxIVhUKhUBgMJSoKhUKhMBhKVBQKhUJhMJSoKBQKhcJgWBjbAGPi5uYm/f39jW2GQqFQNCgOHz6cKqV0r+yzJi0q/v7+HDp0yNhmKBQKRYNCCHG+qs9U+EuhUCgUBkOJikKhUCgMhhIVhUKhUBiMJr2molCYCkVFRcTHx5Ofn29sUxSKUmxsbPD19cXS0vKGz1GiolCYAPHx8djb2+Pv748QwtjmKBRIKUlLSyM+Pp6AgIAbPk+FvxQKEyA/Px9XV1clKAqTQQiBq6trtb1nJSoKhYmgBEVhatTk/0klKjUgKimb9zedQrUNUCgUivIoUakBu6NS+XLnWX47kmBsUxQKhcKkUKJSA+4f4M9N/i68/scJEjPzjG2OQlFrLl++zBdffFGjcxcsWMCVK1cMZsvSpUt5/PHHDTbfjbJz507GjBlT7fOqa++5c+f46aefqn2dGyU9PZ1hw4YRGBjIsGHDyMjIqHTchQsXGD58OB07dqRTp06cO3fOINdXolIDzMwEH04KolgjefHX4yoMpmjwmJKoNHauJSrFxcW1nn/evHkEBwcTFRVFcHAw8+bNq3Tcvffey5w5c4iMjOTAgQN4eHjU+tqgUoprTCvX5rw0qgOvrTnBykNx3NXHz9gmKRoJb/xxgpMXsww6ZydvB+be3rnKz1988UXOnj1L9+7dGTZsGB4eHqxcuZKCggLGjx/PG2+8QW5uLpMnTyY+Ph6NRsOrr75KUlISFy9eZOjQobi5ubFjx45K59+0aRMvv/wyGo0GNzc3/vzzT9LT03nggQeIiYnB1taWxYsXExQUVO680NBQ3njjDczNzXF0dGT37t2cO3eOe+65h9zcXAA+++wzBgwYwM6dO5k7dy6enp6EhYUxYcIEunbtyqeffkpeXh6rV6+mTZs2zJgxAxsbG06cOEFSUhKffPLJVR5Kbm4uTzzxBMePH6e4uJjXX3+dcePGVfn9xcXFMXLkSGJjY5k6dSpz587l1Vdfxc3NjaeeegqAV155BU9PT3766SciIyPp3r079913H87Ozqxfv578/Hxyc3PZvn07H3744VXfP8CyZcv43//+R2FhIX379uWLL77A3Ny8nC1r1qxh586dANx3333ccsstvP/+++XGnDx5kuLiYoYNGwaAnZ1dlfdWXZSo1ILpfVux8fgl3loXycC2bvg62xrbJIWiRsybN4+IiAjCwsLYsmULq1at4sCBA0gpGTt2LLt37yYlJQVvb2/Wr18PQGZmJo6OjnzyySfs2LEDNze3SudOSUnhoYceYvfu3QQEBJCeng7A3Llz6dGjB6tXr2b79u3ce++9hIWFlTv3zTffZPPmzfj4+HD58mUAPDw82Lp1KzY2NkRFRTFlypTSwrDh4eFERkbi4uJC69atmTlzJgcOHODTTz9l4cKFLFiwANB5C7t27eLs2bMMHTqU6Ojoctd95513uPXWW/n222+5fPkyN910E7fddhvNmzev9B4PHDhAREQEtra29OnTh9GjR/Pggw8yYcIEnnrqKbRaLb/88gsHDhwgKCiIjz76iHXr1gG68Nm+ffs4duwYLi4ubNmyhaioqKu+f3d3d1asWMHevXuxtLTkscceY/ny5dx7773MnDmTWbNm0bt3b5KSkvDy8gLAy8uL5OTkq+w9c+YMTk5OTJgwgdjYWG677TbmzZt3lUDVBCUqtcDMTPDBxCBGLtjNC78eY9mDfVVaqKLWXMujqA+2bNnCli1b6NGjBwA5OTlERUUxePBgnnvuOV544QXGjBnD4MGDb2i+f/75h5tvvrl0A52LiwsAe/bs4ddffwXg1ltvJS0tjczMzHLnDhw4kBkzZjB58mQmTJgA6KoPPP7444SFhWFubs6ZM2dKx/fp06f0B7VNmzYMHz4cgK5du5bzoiZPnoyZmRmBgYG0bt2aU6dOXfUdrF27lo8++gjQ7SO6cOECHTt2rPQehw0bhqurKwATJkxgz549zJ49G1dXV44ePUpSUhI9evQoHVPZ+SXfS1Xf/7Fjxzh8+DB9+vQBIC8vrzRktWTJkkrnrYri4mL++usvjh49ip+fH3fddRdLly7lwQcfrNY8laFEpZa0dLHlldGdePn34yzff4Hp/VoZ2ySFolZIKXnppZd45JFHrvrs8OHDbNiwgZdeeonhw4fz2muv3dB8lf2xVdlaZMVxX331Ffv372f9+vV0796dsLAwFi5ciKenJ+Hh4Wi1WmxsbErHW1tbl742MzMrfW9mZlZuvaLidSq+l1Ly66+/0r59++ve37XmmzlzJkuXLuXSpUs88MADVZ5f1gOq6vtfuHAh9913H++99941bfH09CQxMREvLy8SExMrXSvx9fWlR48etG7dGoA77riDf/75xyCiohbqDcCUm1oyONCNdzdEciFNLVgqGh729vZkZ2cDMGLECL799ltycnIASEhIIDk5mYsXL2Jra8v06dN57rnnOHLkyFXnVkb//v3ZtWsXsbGxAKXhr5tvvpnly5cDuswrNzc3HBwcyp179uxZ+vbty5tvvombmxtxcXFkZmbi5eWFmZkZP/74IxqNptr3Gxoailar5ezZs8TExFwlHiNGjGDhwoWlwnf06NFrzrd161bS09NL124GDhwIwPjx49m0aRMHDx5kxIgRwPW/r6q+/+DgYFatWlUazkpPT+f8+avbmowdO5bvv/8egO+//77StaA+ffqQkZFBSkoKANu3b6dTp07XvMcbRXkqBkAIwft3BjFi/m7mrArn54f6YWamwmCKhoOrqysDBw6kS5cuhISEMHXqVPr37w/oFnGXLVtGdHQ0c+bMwczMDEtLS7788ksAHn74YUJCQvDy8qp0od7d3Z3FixczYcIEtFpt6ZrI66+/zv33309QUBC2tralP4RlmTNnDlFRUUgpCQ4Oplu3bjz22GPceeedhIaGMnTo0CrXOa5F+/btGTJkCElJSXz11VflvB2AV199ldmzZxMUFISUEn9//9I1kMoYNGgQ99xzD9HR0UydOpXevXsDYGVlxdChQ3FycipdrwgKCsLCwoJu3boxY8YMnJ2dy801fPhwIiMjr/r+O3XqxNtvv83w4cPRarVYWlry+eef06pVq3JrKi+++CKTJ0/mm2++wc/Pj9DQUAAOHTrEV199xZIlSzA3N+ejjz4iODgYKSW9evXioYceqvb3WBmiKafD9u7dWxqy8+PKQ3E8v+oYc2/vxP0Db7wAm0IRGRlZZbxeYVhmzJjBmDFjmDhxYp1fS6vV0rNnT0JDQwkMDKzz69UFlf2/KYQ4LKXsXdl4Ff4yIJN6+TK0vTvvbzpFbGqusc1RKBRG5OTJk7Rt25bg4OAGKyg1QYW/DIgQgnl3BjHsk13MCQ1nxSP9MVdhMEUTom/fvhQUFJQ79uOPP9K1a1cjWXQ1S5curdF5mzdv5oUXXih3LCAggN9//73S8Z06dSImJqZG12rIKFExMJ4ONrwxrjNPrwjn2z2xPHRza2ObpFDUG/v37ze2CXXGiBEjShfbFVWjwl91wB3dfRjWyZMPt5wmOjnH2OYoFApFvaFEpQ4QQvDu+K40tzLn2dBwijVaY5ukUCgU9YISlTrC3d6aN8d1ITzuMov/anpxVYVC0TRRolKH3N7Nm9FdvViwNYrTl6re7KRQKBSNBSUqdcyb4zpjb2PBs6FhFKkwmMJEMaXS96qfSu24kX4qO3bsoHv37qUPGxsbVq9ebZDrK1GpY1ztrHlnfBciErL4cudZY5ujUFSKKYlKY8cU+qkMHTqUsLAwwsLC2L59O7a2tqXFN2uLSimuB0Z28WJcd2/+92cUwR096OztaGyTFKbMxhfh0nHDztmiK4RU3qwJVD+VptZPpSyrVq0iJCQEW1vDtO5QolJPvH57Z/4+m8azK8NZ+/ggrCyUk6gwHVQ/labVT6Usv/zyC88888w1x1QHJSr1hHNzK94b35WZPxzis+1RPDP8xkpqK5og1/Ao6gPVT6Xx91MpITExkePHjxt0U2ediooQYiTwKWAOLJFSzqvwuTXwA9ALSAPuklKe038WBCwCHAAt0AewBP4qM4UvsExKOVsIMR8Yqj9uC3hIKZ3q6NZqxG2dPLmzpy+f7zzLsE4t6OqrwmAK00P1U2n8/VRKWLlyJePHj8fS0vKac1aHOovBCCHMgc+BEKATMEUIUbFg/4NAhpSyLTAfeF9/rgWwDJglpewM3AIUSSmzpZTdSx7AeeA3ACnl02WOLyw5bmq8dnsn3OyseDY0jILi6veBUCjqAtVPpWn1Uynh559/ZsqUKde8t+pSl57KTUC0lDIGQAjxCzAOOFlmzDjgdf3rVcBnQifxw4FjUspwACllWsXJhRCBgAflPZcSpgBzDXMbhsWxmSXv3xnEjO8OsmBbFC+M7GBskxQK1U+lifVTAd26UlxcHEOGDKn293ct6qyfihBiIjBSSjlT//4eoK+U8vEyYyL0Y+L1788CfYHp6EJiHoA78IuU8oMK878GOEgpn6twvBXwD+ArpbzqTxghxMPAwwB+fn69KlP6+uDFX4+x8lAcqx4dQE8/5+ufoGjUqH4q9Yfqp1I9TKmfSmU13ysqWFVjLIBBwDT983ghRHCFcXcDP1dy/t3AqsoEBUBKuVhK2VtK2dvd3f1a9tcpr4zuSAsHG54LDSe/SIXBFIrGhuqnYnjigZZl3vsCF6sYE69fR3EE0vXHd0kpUwGEEBuAnsCf+vfdAAsp5eFKrns38B8D3kedYG9jyQcTuzH9m/18vOU0r4w2TH9ohcKYqH4q/6L6qRieg0CgECIASED3Yz+1wpi1wH3APmAisF1KKYUQm4HnhRC2QCEwBN1CfglTqMRLEUK0B5z185k8gwLdmN7PjyV7YhneuQV9/F2MbZJCUStUPxVFnYW/pJTFwOPAZiASWCmlPCGEeFMIMVY/7BvAVQgRDTwDvKg/NwP4BJ0whQFHpJTry0w/mcpDX1PQrb/UzUJRHfBSSEd8nZsxJzScK4W1L9GgUCgUxqTOFuobAr1795YlO3GNyT8xady9+B9mDPDn9bGdjW2OwgiohXqFqWJKC/WKG6Rfa1dmDPBn6d/n2Hf2quxphUKhaDAoUTERnh/ZHn9XW+asCie3QIXBFApFw0SJiolga2XBR5O6kXA5j/c2RhrbHEUTw5RK36t+KrXjRvqpADz//PN07tyZjh078uSTT1ZaNqcmKFExIXr7uzBzUADL/rnAnqhUY5ujaEKYkqg0dkyhn8rff//N3r17OXbsGBERERw8eJBdu3bV+tqgqhSbHM8Ob8/2U8k8vyqczU/fjL2N4Qq9KRoG7x94n1Ppp64/sBp0cOnACze9UOXnqp9K0+qnIoQgPz+fwsJCpJQUFRXh6elZ5f1VByUqJoaNpTkfTerGnV/+zdvrInl/YtD1T1Ioaonqp9K0+qn079+foUOH4uXlhZSSxx9/3GDZh0pUTJAefs48MqQNX+48y8iuLRjavurS1YrGx7U8ivpA9VNp/P1UoqOjiYyMJD4+vvT6u3fv5uabb67WPJWhRMVEmX1bIH9GJvHir8fYMnsIjrYqDKaoH1Q/lcbfT+X333+nX79+2NnZARASElIq/rVFLdSbKNYW5nw8qTupOYW8se6Esc1RNHJUP5Wm1U/Fz8+PXbt2UVxcTFFREbt27VLhr6ZAV19H/nNLG/63PZqQLl4M62SYhTSFoiKqn0rT6qcyceJEtm/fTteuXRFCMHLkSG6//fZqf4+Vocq0mECZlmtRWKxl3Od7Sc0pYMvsm3FubmVskxR1gCrTUn+ofirVQ5VpaWRYWZjx8aRuZOQWMnetCoMpFA0F1U9FYbJ08nbgyeBAPtl6hpAuLQjp6mVskxSKSlH9VP5F9VNRmDSP3tKGrSeT+O/qCG4KcMHVzvr6JykU9Yzqp6JQ4a8GgqW5GR9N6kZ2fjH/XR1hsDo9CoVCYUiUqDQg2rewZ/awQDZGXGLdsURjm6NQKBRXoUSlgfHw4NZ0b+nEq2siSM7ON7Y5CoVCUQ4lKg0MC30Y7Eqhhld+V2EwhUJhWihRaYC09bBjzvD2bD2ZxOqwBGObo2gEmFLpe9VPpXbcaD+VF154gS5dutClSxdWrFhhsOsrUakJmmLdw4g8MCiA3q2cmbvmBJcyVRhMUTtMSVQaO6bQT2X9+vUcOXKEsLAw9u/fz4cffkhWVlatrw0qpbhmHF8Ju96HAU9A92lg2azeTTA3E3w4qRshn+7mpd+O8e2MPpUW7VM0PC69+y4FkYbtp2LdsQMtXn65ys9VP5Wm1U/l5MmTDBkyBAsLi9KSMZs2bWLy5MlV3uONojyVmuDgDbZusP5ZmN8Fdn0IeZW7mHVJgFtzXhjZgR2nUwg9FF/v11c0HubNm0ebNm0ICwtj2LBhpf08wsLCOHz4MLt372bTpk14e3sTHh5OREQEI0eO5Mknn8Tb25sdO3ZUKSgl/VR+/fVXwsPDS2tRlfRTOXbsGO+++y733nvvVeeW9FMJDw9n7dq1wL/9VI4cOcKKFSt48sknS8eHh4fz6aefcvz4cX788UfOnDnDgQMHmDlzJgsXLiwdV9JPZf369cyaNYv8/PLefkk/lYMHD7Jjxw7mzJlTKmKVceDAAZYvX05YWBihoaEcOnSIBx98sLSeWUk/lWnTpjFv3jwGDx5MWFgYTz/9NAD79u3j+++/Z/v27eX6qZT9/iMjI0v7qZSU/S8pyDlz5szSnjI30k+lW7dubNy4kStXrpCamsqOHTuIi4ur8v6qg/JUakLrWyBgCJzfC3sWwI63Yc986DUD+j8Gjr71Zsp9/f3ZFHGJt9adZGCgGz5O9e81KQzLtTyK+kD1U2n8/VSGDx/OwYMHGTBgAO7u7vTv3x8LC8PIgRKVmiIE+A/SPS5FwN//g/1fwYFF0HUyDHwKPDrUuRlmZoIPJ3Zj5Ke7efHXY/zwwE0qDKaoFaqfSuPvpwK6cNwrr7wCwNSpUw1Wn0yFvwxBiy4wYTE8FQZ9ZsLJ1fBFX/jpbji/r84v7+dqy0ujOvJXVCo/HbhQ59dTND5UP5Wm1U9Fo9GQlpYGwLFjxzh27FipV1dblKdiSJz8IOR9GPICHPha57l8NxJa9oWBs6HdSDCrGx2f3tePzRGXeGd9JDcHutPSxbZOrqNonKh+Kk2rn0pRUVFp+NLBwYFly5YZLPyl+qnUZT+VwitwdBnsWwiXL4Bbexj4pC48ZmH4vigJl/MYMX83XX0cWT6zL2ZmKgzWUFD9VOoP1U+leqh+KqaElS30fRieOAoTloC5Faz5D3zaDf5eCAVVu8A1wcepGa+O6ci+mDR+/Odqt1ihUNQfTbWfivJU6rPzo5Rw9k9dxti5v8DGUbcG03cW2FW+mFb9S0hmfHeQA7HpbHxqMP5u1Q8NKOqfxuKpNIR+KjWluv1UGgvV9VSUqBirnXDCYZ24RP6h82C6T9VtpnRtU+upEzPzGD5/Nx1a2PPLw/0xV2EwkycyMpIOHTqozD2FSSGl5NSpU6YT/hJCjBRCnBZCRAshXqzkc2shxAr95/uFEP5lPgsSQuwTQpwQQhwXQtgIIeyFEGFlHqlCiAVlzpkshDipP6fuiusYAp9ecNeP8Pgh6D4FwpbDZ71h5X1w8dqZJtfDy7EZr9/emYPnMvhub6yBDFbUJTY2NqSlpakCoQqTQUpJWlraVUkM16POPBUhhDlwBhgGxAMHgSlSypNlxjwGBEkpZwkh7gbGSynvEkJYAEeAe6SU4UIIV+CylFJT4RqHgaellLuFEIHASuBWKWWGEMJDSnn1VtIyGNVTqUh2Euz/Eg5+CwWZus2Vg2ZD66G6PTHVRErJQz8c4q+oVDY8NZg27nZ1YLTCUBQVFREfH3/Vzm6FwpjY2Njg6+uLpaVlueNGCX8JIfoDr0spR+jfvwQgpXyvzJjN+jH79EJyCXAHQoCpUsrp15g/ENgO+EkppRDiA+CMlPKGt5aalKiUkJ8Fh7+DfV9AziVoEaTbSNnpDjCvXspfclY+w+bvprV7c1bNGqDCYAqFwiAYK/zlA5QtJhOvP1bpGCllMZAJuALtACmE2CyEOCKEeL6S+acAK+S/qtgOaCeE2CuE+EcIMdKA91J/2DjoRGT2MRj7GRTlwa8PwsKeur0vRXk3PJWHgw1vjuvM0QuX+fqvmDo0WqFQKHTUpahU9mdxRbeoqjEWwCBgmv55vBAiuMK4u4Gfy7y3AAKBW9AJzhIhhNNVRgnxsBDikBDiUEpKyo3ch3GwsIae98B/DsBdy3XZYRue0xew/ACupN/QNGO7eTOycws+2XKGqCTDpjArFApFRepSVOKBlmXe+wIXqxqjD385Aun647uklKlSyivABqBnyUlCiG6AhZTycIW51kgpi6SUscBpdCJTDinlYillbyllb3d399reY91jZgYdx8CDW2HGBt0C/453dOKy6SXIvHZ1YiEEb4/vgp2NBc+GhlOs0daT4QqFoilSl6JyEAgUQgQIIazQeRZrK4xZC9ynfz0R2K4PZ20GgoQQtnqxGQKcLHPeFMp7KQCrgaEAQgg3dOGwxhPzEQL8B8K0lfDo39DxdjiwWLeR8vdZkHSyylPd7Kx5a1wXjsVn8tWus/VotEKhaGrUmajo10geRycQkcBKKeUJIcSbQoix+mHfAK5CiGjgGeBF/bkZwCfohCkMOCKlXF9m+slcLSqbgTQhxElgBzBHSplWN3dnZDw7w4RF8GQY9HkITq6BL/vD8slw/m/dJssKjA7yYkyQF5/+GUVkomE6vCkUCkVF1OZHU8v+qglX0uHgEl0Byytp4HuTLh25XUi5ApbpuYUMn78LD3sbVv9nIFYWqkqPQqGoPqr2V2PH1gWGPA+zI2DUR5CTBL9M1ZXfP/IjFOvKZrg0t+Ld8V05mZjF5zuijWy0QqFojChRaUxY2cJND8ETR+DOb3QZZGsf16277P0f5GcxvHMLxvfw4fMd0UQkZF5/ToVCoagGSlQaI+YW0HUiPPIXTP8N3AJh66u6jLFtb/DGUDdcmlvx7MpwCoqr3+BIoVAoqkKJSmNGCGgbDPf9AQ/tgDZDYc98HBb15He/lRQkn+F/f0YZ20qFQtGIUKLSVPDpCZO/hycOQ/ep+JxbzXbr5+i85wlOH9llbOsUCkUjQYlKU8O1Ddy+AGYfp6j/bAaZn6D92rFovhsD0dsqTUdWKBSKG0WJSlPF3hPrEa9zfNJe3i6aRu7FU7DsTlg0GI6vAk2xsS1UKBQNECUqTZyBnQPI7fUovXM+5tygD3Tpx78+CAt7wP7FUHjF2CYqFIoGhBKVGlAQHU3aN9+ivdI4fnBfGd0Rd0d77g9rT95Df8PdP4O9F2ycAwu6wM73b7iApUKhaNpcd0e9EKINEC+lLBBC3AIEAT9IKS/Xg311Sk131Kd+tYiUBQswd3bG5f77cZ46FXO7ht0L/u/oVKYu2c/9A/2Ze3tn3cHz+2DvAjizCSxtocMYcGoJdp66qsl2LfTPnmCtmoApFE2FWjXpEkKEAb0Bf3T1tdYC7aWUowxsZ71TmzItV44eJfXLL8nd/Rdmjo643HcvLtOnY+7gYGAr64/X1kTww77z/PJwP/q1dv33g6ST8PdCiNkBOckgK9nbYtkc7D3LCI5nhYcH2LcAW7dqNxtTKBSmRW1F5YiUsqcQYg6QL6VcKIQ4KqXsURfG1ieGqP2Vd/w4qV9+Rc727ZjZ2eF8z3Rc7r0XC2dnA1lZf1wpLGbkgr8A2PjUYJpbV/Ljr9VCXrquFEz2JZ3I5CTpn8u+T4L8ynbsC2judrXg2HmWESX9MWuHGrVSVigUdUttRWU/sAB4BbhdShkrhIiQUnYxvKn1iyELSuZHRpL65Vdkb9mCma0tztOm4nL//Vi4uBhk/vriQGw6dy3ex/S+rXjrjlr+Jy7KLyM4SRUe+mPZ+vfaoqvPt2hWxuvRezqVeULN3cHCqna2KhSKG6a2otIJmAXsk1L+LIQIAO6SUs4zvKn1S11UKc4/c4a0rxaRtXEjwsYG57vvxvWB+7FoCA3B9Ly17iTf7Ill+cy+DGzrVvcXlBLyMir3drIriFBeFQkDzVz0ouNRYc2ngidk46S8H4WiltRKVCpM5Ay0lFIeM5RxxqQuS98XxMSQtmgRmX+sQ1ha4jR5Mq4zH8TS07NOrmdI8go1jP7fXxQUa9k0ezD2NpbGNulfigsht0zILftS5Z5QdhJoCq4+39yq8kSDip5Qcw+wtKn/+1MoGgC19VR2AmPR9YAPA1LQtfp9xsB21jv10U+l8Px5UhcvJnPNWoQQON45AbeHHsLSx6dOr1tbDp/PYNJXf3NXn5a8NyHI2OZUHymhIKuM8FQWhtO/z00FKvl3YOMELfvCba+DZ6d6vgGFwnSpragclVL2EELMROelzBVCHJNSNsBfmvLUZ5OuwvgE0r7+msu//QZS4njHONwefhgrP796uX5NeG9jJIt2xfD9AzcxpF3DCd9VG02RTljKJRwkQVYiRPyqE6ee98HQV8CuEX8PCsUNUltROQ4MB74HXpFSHlSiUnOKEhNJW/INl0NDkRoNjmPG4DrrEawDAurVjhshv0jDmIV7yMkvZvPTN+PYzITCYPXFlXTY9QEc/FqXOHDzs9D3URUaUzRpatv58U10+1PO6gWlNaDqpdcQSy8vWrz6X9ps24rL9Olkbd5MzOgxJDz7HAXRptWN0cbSnI8ndSMlp4C31p00tjnGwdYFQubBY/9AwGDY9jp83gciflPFNxWKSlA96og3iLcAACAASURBVI3co744LY30774j/aefkXl52A8fjtujs7Dp0MGodpXlo82n+WxHNF9N78WIzp6Ippw9FbMTNr8CSRHQsh+MeBd8exnbKoWiXqlt+MsXWAgMRLeauQd4SkoZb2hD6xtTEJUSijMySP/+ezJ+XIY2Nxe74GDcHn2UZl06G9s0Coo1jPtsL6cuZWNpLnC3s8bd3hp3exv9szUe9tblXrvZWWNjaW5s0+sGrQaOLoPtb+sy0bpOhtvmgqOvsS1TKOqF2orKVuAn4Ef9oenANCnlMINaaQRMSVRK0GRmkv7jMtJ/+AFtVhbNh9yM+6OP0qx7d6PalZyVzx/HEknJLiAlu4Dk7HxSsgtIzSkgLbew0kiQg40FHg42uNtZ4+FgXSpGutc2pQLkZGvZML2fgmzYMx/+/gyEGQx4AgY+peqgKRo9ta79JaXsfr1jDRFTFJUSNDk5ZCxbTvrSpWguX6b5gAG4/ecxbHuZXqilSKMlPbewnNjoXld8zie/SHvV+ZbmAje78t5OiRdUekwvSCbp/Vy+oFtrifhVt/cl+DXoNgXMVBFwReOktqKyDVgK/Kw/NAW4X0oZbEgjjYEpi0oJ2txcMn75hbRvv0OTlobtTTfh9thj2Pa9qcH9dS+lJKeg+CrRSckpIDlL96z7LP+a3o9ObKoKvemOOzWzxMysnr+fuAOw+WWIPwgtgmDke+A/qH5tUCjqgdqKih/wGdAf3ZrK38CTUsoLhja0vmkIolKCNi+PyytXkrbkG4pTUmjWsydujz1G84EDGpy43AjFeu+nVHjKekFlRCg5q4C8oqurJpd4P1d5O/pwXNnjBvV+pNR5LFvnQla8rl3AsDd1bZwVikaCwcq0lJlwtpRyQa0tMzINSVRK0BYUcHnVKtK+XkLxpUvYBAXh9tij2A0Z0ijF5UYo8X6Ss/JLvZ3yYqR7TsstqNT7sbexKOPt2JQToR5+TrR2r8EaSVEe7Ptct+ZSXAB9H4Gb50Azp9rfsEJhZOpCVC5IKU13K/gN0hBFpQRtYSGZv68mbfFiihISsOnUCddHZ2EfHIxQsfxKKef95BSQUibkVnEt6EqhzvsxNxPcP8Cf2cPaYVdZK4DrkZ0E29/SZYs1c4ahL0Ov+1VPGUWDpi5EJU5K2bLWlhmZhiwqJciiIjLX/kHq4kUUnb+Adbt2uD06C/vhwxHmJrio3UDILSgmMTOfb/bE8svBC3jYW/Pf0Z0YE+RVM4/w0nHdekvsbnBrD8PfhsBhqmKyokGiPJUqaAyiUoIsLiZrwwZSv1pEYUwMVm3a4DZrFg6jQpS41JKwuMv8d/VxIhKyGNTWjTfGdaZNTUJiUsLpjbDlv5B+FtrcCsPfUcUqFQ2OGomKECKbSku3IoBmUsoG7783JlEpQWo0ZG/eTOqXX1EQFYVVq1a4zpqF45jRCMsmWLvLQGi0kp/2n+eDzafJL9Lw0ODWPH5rW2ytavDPoLgQDi6BXfN0e11UsUpFA8PgnkpjoTGKSglSqyV72zaduERGYunri+vDD+F0xx0IK9Ulsaak5hTw3oZT/HokHh+nZrx2eyeGd6ph6Zor6bDrfZ3AWNrC4Geh36NgYW14wxUKA2I0URFCjAQ+BcyBJRW7RQohrIEfgF5AGrqOkuf0nwUBiwAHQAv0ASyBv8pM4Qssk1LOFkLMAD4EEvSffSalXHIt+xqzqJQgpSRnx05Sv/yS/OPHsfDywvWhmTjdeSdm1urHq6YciE3ntTURnLqUzdD27rwxtgt+rrY1myw1Cra8Cmc2glMrXQpyp3FqvUVhshhFVIQQ5sAZYBgQDxwEpkgpT5YZ8xgQJKWcJYS4GxgvpbxLCGEBHAHukVKGCyFcgctSSk2FaxwGnpZS7taLSm8p5eM3amNTEJUSpJTk7tlD6hdfknf0KBYeHrjOfBCnyZMxs1Fl3GtCkUbL93+fY/7WMxRpJY/d0oZZQ9rUfN/L2R269ZakCPDrDyPeAR/Tq6CgUNS29H1NuQmIllLGSCkLgV+AcRXGjEPXpwVgFRAsdHGE4cAxKWU4gJQyrRJBCQQ8KO+5KKpACIHd4MG0+mk5fku/w6pVK5LefY/o24aR9u13aK9cMbaJDQ5LczNmDm7Nn8/ewvBOnizYFsWIBbvZcTq5ZhO2GQqP7Ibb/wdpZ+HrW+G3RyAz4frnKhQmQl2Kig8QV+Z9vP5YpWOklMVAJuAKtAOkEGKzEOKIEOL5SuafAqyQ5V2tO4UQx4QQq4QQlaY8CyEeFkIcEkIcSklJqdmdNWCEEDTv149WP/5Aqx9/wKZdIMkffEB08G2kLlqMJifH2CY2OFo42vDZ1J4sn9lXt6/lu4M88uMhEi7nVX8yM3PodR88eQQGPQMnfoeFvWDHu1CYa3jjFQoDc11REUJkCyGyKjzihBC/6xt2VXlqJccqxtqqGmMBDAKm6Z/HCyEq1hq7m3/rkQH8AfjrO1Ju418PqPzkUi6WUvaWUvZ2d2/a2Ta2ffrg9+23tPr5J2y6diFl/nyig28j5fPP0WRlGdu8BsfAtm5seupmnh/Znt1nUrnt4118sTOawuKri2heF2t7XTn9xw9C+xDdgv7CXhD2E2hrMJ9CUU/ciKfyCTAHnVfhCzwHfI0unPXtNc6LB8p6C77AxarG6NdRHIF0/fFdUspUKeUVYAPQs+QkIUQ3wEJKebjkmD5EVqB/+zW6xX/FDWDbowd+ixfjH7oS2169SF34GdG3BpP86acUZ2QY27wGhZWFGY/d0patz9zM4EA3Pth0mpBPd/N3dGrNJnRuBZO+gwe3goMPrH4Uvr4Fzu01qN0KhaG4EVEZKaVcJKXMllJmSSkXA6OklCsA52ucdxAIFEIECCGs0HkWayuMWQvcp389EdiuD2dtBoKEELZ6sRkClO1nO4XyXgpCCK8yb8cCkTdwb4oyNOvalZZffE7A77/RfMAA0r78irPBt5H88ccUp6cb27wGha+zLYvv7c13M/pQpJFMXbKfJ38+SnJWfs0mbHmTTlgmLIHcNFg6ClZMh/QYwxquUNSSG6lSvA+Yj24hHXQ//s9IKftdr6+KEGIUsABdSvG3Usp3hBBvAoeklGuFEDbomn/1QOeh3C2ljNGfOx14CV04bIOU8vky88agE7ZTZY69h05MivVzPVr288poStlfNSH/zBnSvlpE1saNCBsbHEaPolnnzlgHBmIdGIi5o6OxTWwQ5Bdp+HLnWb7cdRYrczOeHtaO+/q3wsK8hkuaRXmw7zP4az5oClWxSkW9U9vS963R7TXprz+0D3ga3X6QXlLKPQa0tV5RonJjFMTEkLZoEdnbd6DNzi49buHhoROYdu1Khca6TWvMbGu4X6ORcz4tl7lrT7DzdAodWtjz9h1d6O3vUvMJsy/pi1UuB1sXuOUlVaxSUS+oHfVVoESlekgpKU5KoiAqioIzUfrnMxScPYss0C9nCYFly5Z6kWmLdWAgNu3aYeXvr8rEoPsON59I4s0/TnAxM587e/ry0qgOuNnVYiNqYjhsfgXO/aUrVjniHV2xSoWijqitp+ILLAQGogtF7QGeklLGG9rQ+kaJimGQGg1FcXHkR+mFRv8ojD0HGv32IktLrP399Z5NYKlnY+nr2yRL9V8pLGbh9miW/BVDM0tz5ozswNSb/DCvabdKKeH0Bn2xyhhoE6wTF4+OhjVcoaD2orIV+And2gfAdGCalLLB/ymkRKVu0RYWUhgb+69Xo38Uxf/794ho1gzrNm2uCqNZeLg3iaZj0ck5vLYmgr/PphHk68hb47rQrWUt1kYqFqvsNUNXrLK5m8FsVihqKypXLcZfb4G+oaBExThoc3MpOHtWFzrTC01+VBSalH/Tbs0cHUvDZ9aBgdiUJAc4Nb7FaCklfxxL5O11J0nJKWDKTX48P6I9Tra1KPx5JR12ztMJjFVzuPk56DtLFatUGITaiso2YCn/pvBOAe6XUlbcjNjgUKJiWhRnZJTzaAqioik4c6Z8coC7e/nEgHaBWLdp0yiSA7Lzi1iwLYqlf5/DsZklL47swMRevpjVNCQGkHJGFxKL2gzO/rpilR3HqmKVilpRW1HxAz5Dl/0lgb+BJ6WUFwxtaH2jRMX0qTQ5ICqKgujo8skBvr7/Ck2J2Pj7N8gy/5GJWby2JoKD5zLo6efEW3d0obN3LdO3z26Hzf+F5BPgN0BfrLLn9c9TKCqhLjo/zpZSLqi1ZUZGiUrDRWo0FMXHk18mhHZVcoCFBdYB/uXFpiQ5wMS7YUop+fVIAu9tiCTjSiH39vfnmeHtcLCpRQadVgNHfoAd70BuCnSbAre+Co4VS/IpFNdGtROuAiUqjQ9dcsC5f9OdK0sOsLG5OjmgXSAWHh4mlxyQeaWIj7acZtn+87jZWfPKqI6M6+5dOzvzs2DPJ7DvCxBmMPApGPikbu1FobgB6kJU4qSUlVYBbkgoUWk6lCYHVAijFZepVG3m4FBuf411YCDNOnfGrLnxf2yPx2fy3zURhMddpl9rF94a14VAT/vaTZpxHrbN1VVCtveC4LkQdBc0oRTv7PwiIhKy8HezxcuxmbHNaTAoT6UKlKgorpccYOHujs8nH2Pbp4+RLQWtVvLLwTje33SK3IJiHhwUwJPBgTS3ruUO+gv7YfNLkHAYvLrDmE8aZXMwKSXxGXkcOp/O4fMZHD5/mdOXstBKXd7CTf4ujOvuw6iuLWqXedcEqJGoCCGyubpUPejK1TeTUjb4WhBKVBSVIaWkODmZ/BMnSX7/fQrj4nB/6ilcH5ppEhs103MLeX/jKVYcisPL0YZXx3QipEuL2oXEtFqIWAXbXofcVJiwCDqPN5jNxqCgWMOJi1kcOZ/BoXMZHL6QQUq2LrnDztqCHn5O9PRzpltLRyISslgdlkBMSi6W5oIh7Ty4o4c3wR08aWZl2utvxkCVaakCJSqK66HJySHx1VfJ3riJ5jcPxvv997FwvlZx7vrj8PkMXl0dwcnELAYHuvHmuC4EuNUyVJebBr9Mhbh/4LY3dOstJrbOVBVpOQU6D+RCBkfOZxAen1nay8bPxZZerZzp2cqZXn7OtG9hf1X1AiklJy5msSYsgbXhF0nKKqC5lTkjurRgXHcfBrZxrXkR0EaGEpUqUKKiuBGklGT8/DPJ783D3NUVn08+wbZnD2ObBUCxRsuyf87z8ZYzFBRreWRIa/4ztC02lrX467ooX9e35cRv0PsBCPnQ5IpUarWS6JQcnQdyPoMjFzKITdV1xrQ0F3TxcaSXnzO9/Z3p6eeMh4NNtebXaCX7Y9NYc/QiGyISyc4vxs3OijFB3ozr7k33lk4ml9RRnyhRqQIlKorqkBdxgoTZsym6dAmPZ57B5f4ZJvPDkpydz3sbTvH70QR8nZvx+u2dua2TZ80n1Gph+5uwZz60HaZrFGZdy8SAWpBbUEx43GUOn8/gkF5EsvOLAXBtbqXzQPSPrj6OtRPVChQUa9hxKoW14Qlsi0ymsFhLK1dbxnXzZmx3H9p62BnsWg0FJSpVoERFUV00WVlcfPllcrb9iV1wMN7vvmNSfWX+iUnj1dURRCXncFtHD+be3pmWLrWoNnB4Kax7Bjw6wbSV4OBtMFurQkrJxcx8Dp1L54g+nBWZmI1Gq/utaudpR69WLqUi4u9qW2/inpVfxOaIS6wNv8je6FS0Err4ODCumw+3d/OmhWP1PKKGihKVKlCioqgJUkoyfviBpA8/wtLTE58F82nWtauxzSqlSKPlu72xLNgWhUYreeLWtjx0c2usLWr413v0Nlg5Q+epTFsJLQx7r0UaLScvZukzsnSPS/oOmbZW5nRv6VQqID1aOuNoaxotFJKz8ll3LJE1YQmEx2ciBPQLcGVcd29CuniZjJ11gRKVKlCioqgNeWFhxD/zDMUpqXg+/zzO06eZTDgMIDEzj7fXRbL+eCKt3ZrzxrjODA50r9lklyLgp8mQnwmTvofA22psV0ZuIUcu/Csg4fGXyS/SLaj7ODUrFZBerZzp0MK+QSyOx6TksDb8ImvCLhKbmouVuRm3tHfnjh4+3NrBw6DhOFNAiUoVKFFR1JbijAwSX3yJnF27sB8xAq+338Lc3nhrD5Wx+0wKc9eeIDY1l9FdvfjvmI412+iXdVEnLEknYfTH0Pv+654ipeRsSq4urVe/P+Rsim5B3cJM0NnboTSU1bOVU4PfgCil5HhCJmvCLvJH+EWSswuwt7bQZ5B5M6CNW8175pgQSlSqQImKwhBIrZb0b78lef4CLH198F2wAJuOptUcq6BYw9e7Y1i4PRpzM8FTwYE8MCgAy+p6AQXZsOoBiNoCA2frduGX2buTV6ghPF63oF6yHnL5ShEATraW9PLTpfX2buVMkK9To94DotFK/olJY/XRBDZFXCK7oBh3e2vGBHlxR3cfgnwdTcqzrQ5KVKpAiYrCkFw5fJiEp59Bc/kynq+8gtPkSSb3oxGXfoU3/jjJtsgk2nna8ea4LvRr7Vq9STTFsHEOHPqWvHZj2dnxLQ7E67yRExezKNYvqLf1sKOXn3Pp/pA27s1N7vuoL/KLNOw4lcyasItsP5VMoUZLgFtzxnbTpSi3dm9YGWRKVKpAiYrC0BSnp3NxzvPk7t2Lw5gxeL3xuknUDqvItpNJvP7HCeIz8hjfw4eXRnXAw/7amUvFGi2nLmXr1kLOpdMuZimPF3/PIW07HpfP0crXj97+/y6oOzdXpU4qIzNPl0G2OiyBfTFpSAlBvo6M7ebN7d288azmnhpjoESlCmoqKmui17A8cjm+9r742vniY+eje23vi3dzbyzNG2/Wh+L6SK2WtEWLSFn4GVb+/vgsmI9Nu3bGNusq8go1fLEzmkW7YrC2MOPZ4e2Y3q9V6cJ4Zl4RR/S70w+fzyAs7jJXCnVtBVo42NDL35k7rQ9xy8n/Ihx9ENNWgWsbY95SgyMpK58/9Av8xxN0GWQD2rgyrpsPI7u2qF2rgzpEiUoV1FRU/jz/J6FRoSRkJ5CQk0CRtqj0M4HAs7mnTmjsfPGx1z2XCJBbM7cmGwJoauT+s5+E555Dm5NDi9dew2mCadbSik3N5bU1EfwVlUonLwe6tXTk8PkMziTlAGBuJujk5VAaxurdyhlvpzIL6nEH4Oe7QUq4+ydo1d9Id9KwiU4uySBL4HzaFawszLi1va4G2S3tTSuDTIlKFRgi/KWVWpKvJJOQk0B8dvxVz8l5yeXGW5tb42PnU+rdlHo5euFpbml6oRJFzSlOSSHh2ee4cuAAjuPH0+K1VzFrZnoZTlJKNkVc4u31kWTnF5WKR89WznTzdbp+JeT0GFg+CS5fgPFfQZc768fwRoiUkvD4TNaEJfBHeCKpOQXY21gQoq9B1q+1q9EzyJSoVEF9rKkUaApIyEkgITuB+Jz40ucS4ckpyik33tna+SrBKXlu0bwFlmam6Q4rqkZqNKR+/jmpX36Fddu2+Hy6AOvWrY1tVqVIKZESzGryo3UlHX6ZBhf+1mWFDXq6wRSjNFWKNVr2xaSx+uhFNp+4RE5BMR721tzezZs7uvvQxcfBKJEPJSpVYOyFeiklmQWZJOQkEJcTd5XwJOYkUiyLS8ebC3NaNG9RXmz0Ho6PnQ8uNi4qtGbC5OzZy8U5c9AWFOD1xhs43j7G2CYZnuICWPMfOB4KPe+F0Z+AWmM0CPlFGv6MTGZNWAI7T6dQqNHS2r0547r5MLa7d+0rVFcDJSpVYGxRuR7F2uJyobW47Djda73wpOWnlRvfzKLZVeG0EuHxtvPG1rIWNaAUBqEoKYmEZ54l7/BhnCZPxvOVlzGztja2WYZFStjxDuz+ENrcqtuBb+NgbKsaFZlXitgYkcjqsAT2x6YjJXTzdWRcdx/GdPO6biZfbVGiUgWmLirX40rRFS7mXNSJjF54yobW8orzyo13tXEtTRzwsfOhpX3LUhHytPXE3Mx0FgIbM7KoiJRPPyVtyTdYd+yI74L5WLVqZWyzDM+RH2HdbHBrr6sZ5uhrbIsaJYmZeaUZZCcuZmEmYGBbN8Z282ZklxbY10EGmRKVKmjoonItpJSk56eXC6eVTSJIzE1EK7Wl4y2EBV52XuUy1nzsfWhppxMeR+uGu/vXVMnesYOLL74ExcV4vfMODiNHGNskw3N2B6y8F6yaw9SV4BVkbIsaNdHJ2awJ0wnMhfQrWFuYEdzRg3HdfbilvXvNi4pWQIlKFTRmUbkeRdoiLuVeqjRjLT4nnssFl8uNt7O0o4NLB+5qfxfBrYJVwoCBKEpIIP6ZZ8gPP4bztGl4vPA8ZlaNbNNg0kldZlj+ZZj4HbQbbmyLGj1SSo7GXWbN0QTWHUskLbcQBxsLRnX1Ymx3b/oFuNYsGUOP0URFCDES+BQwB5ZIKedV+Nwa+AHoBaQBd0kpz+k/CwIWAQ6AFugDWAJ/lZnCF1gmpZxdZs6JQCjQR0p5TcVoyqJyPXIKc0rXb+KzdY+9F/cSlx2Hh60Hd7e/m4ntJuJsYxqtdRsysrCQ5I8/Jv37H7Dp0gWfBfOx8m1koaKsRPj5Lrh0HEZ9CH1mGtuiJkOxRsue6FTWhukyyHILNbRwsOHVMZ0YHeRVozmNIipCCHPgDDAMiAcOAlOklCfLjHkMCJJSzhJC3A2Ml1LeJYSwAI4A90gpw4UQrsBlKaWm4o0BT0spd+vf2wPrASvgcSUqhkWj1fBXwl8si1zG/sT9WJlZMbr1aKZ1nEZ7l/bGNq/Bk7V1K4kvvwJC4P3eu9gHBxvbJMNSkKMvRrkZBjwBt71Zrhilou7JK9SwLTKJNWEXeXBQAP3bVLPumx5jiUp/4HUp5Qj9+5cApJTvlRmzWT9mn15ILgHuQAgwVUo5/RrzBwLbAT+pvwkhxAJgG/Ac8JwSlbojOiOa5aeWs+7sOvI1+fRp0YdpHadxi+8tasG/FhTGxZHw1GzyT57EZcYMPJ59BmHZiEKNmmLY9CIc/Bo6jYPxi8DS9DaDKq7NtUSlLv9M8AHiyryP1x+rdIyUshjIBFyBdoAUQmwWQhwRQjxfyfxTgBVlBKUH0FJKue5aRgkhHhZCHBJCHEpJSanJfSmAts5tmdt/LtsmbePpXk8Tlx3H7B2zGf37aL4/8T2ZBZnGNrFBYtWyJa1+/gnnqVNIX7qU8/fcS1FiorHNMhzmFrrw14h34eRa+P52yE01tlUKA1KXolLZKlBFt6iqMRbAIGCa/nm8EKJiLOBu4GcAIYQZMB949npGSSkXSyl7Syl7u7vXsAueohRHa0ce6PIAGyds5JNbPsHT1pOPDn3EsFXDePuft4nJjDG2iQ0OM2trWrz2Gj6ffExBVBSx4yeQs3t3nV/3fNZ5lhxfwsrTK8kpzLn+CTVFCOj/H5j8g26NZUkwpEbV3fUU9Yqphr/uAkZKKWfox70K5EspP9S/7waESinb6d87AmeBkn8JLYB0YOy1QmAq/FU3RKZFsjxyORtiN1CkLWKA9wCmdZzGIJ9BmAkVQ68OBbGxJMx+moLTp3F9+GHcn3wCYXGdOlzV4FLuJTaf28yG2A2cTCtd7qSZRTNCAkKY1G4SnV071106efwh+Oku0BbrilH6D6yb6ygMirHWVCzQLdQHAwnoFuqnSilPlBnzH6BrmYX6CVLKyUIIZ+BPdF5KIbAJmC+lXK8/bx5QIKWcW8W1d6LWVIxOWl4aq86sYsXpFaTkpdDKoRVTOkzhjrZ3qMKZ1UCbn0/SO+9wOXQVtr174/3xx1h6etR4vvT8dLac28LG2I0cST4CQBfXLowMGMkI/xGk5qUSeiaUjbEbySvOo4NLBya1m8SogFHYWdVBM6n0WF2b4oxzMO4LCJpk+GsoDIoxU4pHAQvQpRR/K6V8RwjxJnBISrlWCGED/Aj0QOdZ3C2ljNGfOx14CV04bIOU8vky88YAo6SUp6q47k6UqJgMRZoitpzfwvLI5RxPPU5zy+aMbzueqR2m0tKhpbHNazBkrl1L4tzXMWvWDJ+PPqT5gAE3fG5WYRbbL2xnY+xG9ifuRyM1tHVqS0hACCP9R+Ln4HfVOdmF2WyI2UDomVBOZ5ymmUUzRgWMYlK7SXRy7WRY7yUvA36ZDuf3wK3/hcHPqWKUJoza/FgFSlTqn2Mpx1gWuYyt57aikRqG+A5hWqdp9G3RV+3YvwEKoqOJnz2bwrMxuD32GG6PPYowrzzbLq84j11xu9gYu5G/Ev6iSFuEr50vIQEhhASEEOgceEPXlFISkRpB6JlQNp3bRF5xHh1dOjKx3URGtx5tOK+zuADWPgHHVkCP6TBmgSpGaaIoUakCJSrGI/lKMitOr2DVmVWk56fT1qktUztOZUzrMTSzUCmm10J75QqX3niTzDVrsO3fD58PP8TCzQ3QeYV7L+5lQ+wGdsbtJK84D49mHowIGEGIfwhd3LrUSryzC7NZH7Oe0DOhnMk486/30l639lJrpISd78Gu96H1LbrFfBvH2s+rMChKVKpAiYrxKdAUsDF2I8sjl3Mq/RQOVg7c2e5OprSfgpddzXb7NgWklGT+9huX3nwLM3t7Lr/yIOsdY9l6fivZhdk4WTsxrNUwQgJC6OnR0+B7h6SUHE89rvNeYjeRr8mno0tHJrXXrb3U2ns5uhz+eBLc2ulqhjmpMKkpoUSlCpSomA5SSo4kH2F55HL+vPAnAMF+wUzrOI2eHj1VaKwCWqnlWMox9uxaTpcFG3FP1/DbLdbk3R1CSJtR9PPuV2/12bIKs0q9l6iMKGwtbBnV+t+1lxoTsxNW3AuWNjB1BXj3MJjNitqhRKUKlKiYJhdzLvLL6V/49cyvZBVm0dGlI1M7TiUkIARr80bWe6QaSCk5lX6Kjec2sil2E4m5iVibWxPsNoDJq9Ow3XmE5oMH4/3B+1g4139NNiklx1KPEXo6lM3nNpOvyaeTaycmtZtESEBIzbyX5EhdMcorabpiNGDUBQAAGWBJREFUlO1HGt5wRbVRolIFSlRMmytFV1gXs46fIn/ibOZZXGxcmNRuEne1vwt326azcTU2M5aNsRvZGLuRc1nnsBAW9PfuT0hACENbDsXOyg4pJZdXrCDpnXcxd3XF55OPse3Z02g2ZxVmse7sOkLPhBJ9ORpbC1tGtx7NxHYTq++9ZCfpilEmhkPIB3DTQ3VjtOKGUaJSBUpUGgZSSv5J/IefIn9iV/wuzIU5w/2HM73jdLq6dzW2eXXCxZyLbDq3iY2xGzmVfgqBoE+LPowMGMkwv2E42ThVel7eiRMkzH6aoosX8XjmGVweuN+ooUMpJeEp4YSe0XkvBZoCOrt2ZmK7iYwKGHXj3UgLc+HXmXB6A/T7Dwx/C1SNOaOhRKUKlKg0PC5kXeDnUz/ze/Tv5BblEuQWxLSO0xjmP6zB93hJzUtl87nNbIrdRFhKGABBbkGEBIQw3H84HrY3tuFRk51N4suvkL11K3a33or3e+9i7mj8DKrMgkzWxaxj1ZlVRF+Oprllc0YH6LyXjq4drz+B9v/t3Xl0VPd99/H3VxoJSSAJCS2WBRhhgyGxWURsTIKLiRsQSwi1Wcwm8jxpc9LayWkSO6nd7H3a5ml66qTPk6S1GzdmMRDkOBDbQAjGgHdAMruRjRYsiLUwkhBIaJn59o87iAFrhJBm0fJ9naNzNDN35n5/XKSPfr/fvb/rgR1PwNv/AePmwwNPQ6zdIjsSLFQCsFDpuy62XuR3H/yODe9toPx8ORnxGSy5fQmLb19MalxqpMvrsvrmenad3sXLpS+z/6P9eNXL2JSx7RclDk/s3n1VVJXateuo/MlPiElPJ/unTxI/oXfcdbGj3ssdw+5g8e2LyRuVd/3ey1u/hO2PQ3YuLNsIQ7q/usCA0tYMFfudEyBK9sC934Db53TroyxUArBQ6fu86uW1M6+x/sR63jj7BrFRscwdPZcV41cwLnVcpMvrUGNrI69++CrbSrfx2tnXaPO2MTJxJHk5eczNmcutQ28N2r6aDh92hsOqq8l87DFSVq3sVWfSXe69bD65mVP1pxgcM5j5o+ezaOyizo/fey9BwZecQFlRAOljw1d0X+H1wkeHoXSPEyLlb0BbE0gUZE9xVi3o5okPFioBWKj0LyV1JTz33nNsPbWVprYmpmROYcX4FcwcMRNXVPAWYeyOFk8L+87sY3vpdvZU7KGprYnMhEzyRuUxJ2dO8Jc98eOpq+Ps409wYfduEmfNIusf/w/RiYkh2Vd3qSrvVr/bfuZYi7eFO9PuZNHYRYF7L2cOOotRelp8i1FOD3/hvYkquEt8IfIqlO6DJrfzWvo4yJnhXFA66jM9vqDUQiUAC5X+qb65vn1o7MyFM2QNzmLZuGU8MOYBkgeFb26hzdvGO396h5dLX+aV06/Q0NpAyqAUZo2axZycOUzOmBy2VZtVFfcz/03Vv/0bMdnZDP/pk8R9ogfXkIRQfXM9vz/1ezYXb6akvoQhMUOYN3oei8cu/vgdRmvLnVOO3SXwhZ/DxKWRKTpSLlRB6V4o2Q0le6H+tPN8UrYTIDkzIOfPICm4FxJbqARgodK/ebweXv3wVda/t579H+0n3hXP/NHzWTF+RVCHmPx51UtRVRHbSrexs3wn7ktuhsQM4f6R9zMnZw5Ts6ZGtNfUWFjIma9/A09tLZlPPM7QpUt71XCYP1WlqKqIguKC9t7LhLQJLBq7iNmjZl/pvTTVwaaVULYP7nsCZnyr/y5G2dzgDGNdnhep8i36HpfshEfODBg9E4bdGtJ/AwuVACxUBo6T7pOsP7Gel0peosXbwrSsaawYv4J7h9/b496CqnLcfZxtJdvYXradysZK4qLjmDFiBnNy5jA9e3qvumizze3m7Le+zcXXXiNp3jxu+uEPiR7Su29FUN9cz9ZTWykoLui499LW4izrcmgDTFrhLEbpio102T3X1gJnDlwJkTMHnHvPuOJg5D1XhrSyJob1FGsLlQAsVAae2ku1FBQXsPHkRqoaqxiROILl45az8LaFN3yvkFN1p9hW6gRJ+flyXFEupt88nbycPGaOmNn1azAiQL1ezj31NNX//u/EjhxJ9s9+RtztvX+y+/JyPpuLN7OzbKfTe0mfwKIxi8gbNZv41/8fvPpPzl/tS9ZCfMfX8/RaXi9UHr0yL1L+JrRedCbXb558JURGTHWWr4kQC5UALFQGrlZvK7vKd7HuxDoOVR9icMxgFt62kGXjlnFL0i0B31fRUNF+UWJxbTFREsVdN93F3Jy53D/y/rDO2QTDxbff4cyj38TbcIGbvvtdhj74QKRL6rK6S3VO7+X9AkrrS0mMSXR6LyQy9g8/coaAlv8GUgIfz17BXeo3ub7XWZIGnMU0R9/nBMmo6b0qIC1UArBQMQBHa46y/sR6tpdtx+P1cO/we1kxfgXTsqYhIlQ3VrOjbAfbyrZxuPowAJPSJ7XfKTEtPi3CLeiZtpoazjz6GI1vvUXywoXc9L3vEpXQe3tZ11JVDlYepOD9gvbey8SkHBadPsbslijil29yrmnpLS7WXAmRkj1QV+48n5h1JURGz4CkmyNYZOcsVAKwUDH+qhur2Vy8mU0nN+G+5GZ08mjS4tPY/9F+FGVc6jjm5Mxh9qjZZA/JjnS5QaUeDzW/+CU1v/gF0amppCxbRspDS9vv09JX1F2qY8upLRQUF1B2voxEL8xvbGLxtMcZk/ulyBTVfAFOv3klRCqPOM8PSoace68MaaWN6TMnGFioBGChYjrS4mlhR9kONry3gcbWRmaNmkVeTh6jk0dHurSQazx4kHNPPc2FPXuQmBiS5s0jdXU+ceO7sIxKL6KqHKg8QMHx9ew8vYtWgUlxN7FoyiPMHjWbOFcI5yM8rVBx4EpvpGK/M7kePQhGTr1yhlbWRIiO7PVT3WWhEoCFijEday4ppXbdOupeeAFtaiLhrrtIXZ3PkJkzA96+uLeqbTjL1i1fpKDpNGWxMSTGJvL50Z9n8djF3JZyW893oAqVx/wm19+AlguAwM2TrvRERt4DMf3jrqYWKgFYqBjTOU99PXUFz1O7fj2tZ88SM3w4KStXMPTBB3vdVfmd8nrQHd/hwKFfsTl7LH+kkVZvK5PSJ7H49sXMHDGTxNgbaE9t+dWT6xerneeH3Xb15HpC31mH7kZYqARgoWJM12hbGw27XsG9Zg1NBw8SlZBA8oMPkrpyBbG39PKzq/y9/RRs/zburDvZOmUxBad3UH6+HEEYmzKW3Mxc5ysj9+pVoS+eg7K9V+ZFakud54dkXj25nty9BUD7GguVACxUjLlxTUeO4l67hvPbtkNbG0Puu4/U1fkkTJ3aa6/Ov8rJbVDwvyEhDV3+Gwpp5O0/vU1hZSGHaw7T1NYEwPBBw8iNSiC3vobcqlOMam1DYhOvnlxPv73PTK4Hk4VKABYqxnRfa1UVdRs3UrthI57aWgaNHUvq6nyS5s8nalDvWUGgQ2eLnMUoWy/B0rVwy6fhTCGtJbt5r3QnhXUfUDjIRVFcHLXRzooLqTGJTL7pLnIzp5Cbmcu41HERX6g0UixUArBQMabnvM3NnH/xRdzPrqG5uNg5JfmhpQx96CFiMnrxvU7qTsP6JXDufXDFQ0sDIJA1oX1IS0fcQ+mlSgorCymqKuJg5UHOXDgDQLwrngnpE5iS4YTMnWl39upVFILJQiUACxVjgkdVaXz7bdxr1nJh925wuUiak0dq/mri7/hkpMvr2KV62Pl9Zwjr8oq+15lcr7xY2R4wRVVFFNcWoygucTF+2HgmZ0xun5dJiUsJU0PCy0IlAAsVY0Kjpbwc97r11D//PN7GRuKnTCE1P5/E+z+LuPrXkNH5lvMcqjpEYVUhhZWFHKk5Qqu3FYCc5BxyM65M/mcPye4b807XYaESgIWKMaHlaWig/re/xb12Ha0VFcTcfDMpK1cydNGDRCclRbq8kGj2NHOs5lh7yLxb9S4NrQ0AZCRkXBUytw29jegwri4cLBYqAVioGBMe6vFwYfdu3M+uoXH/fiQhgaELF5KyaiWDcnIiXV5IebwePqj7gMKqQooqizhYdZCqxioAEmMSmZQxqT1kPpn2yV51m4RALFQCsFAxJvwunTiBe81azr/4ItraypAZM0jJX8XgT3+6XwwNXY+qcvbiWQorC9vnZUrqSwCIjYrljrQ7yM3MZXLGZCZlTCIptvf16CxUArBQMSZy2mpqqN24idoNG/CcO8egMbeRsmoVyQsWEBUXuXuFRIL7kpuiqiKKKosorCrkxLkTtGkbgjAmZQy5GblMyZzC5IzJZA7OjHS5kQsVEckDfgZEA/+lqj++5vVBwBpgCnAOWKqqZb7XJgD/CSQBXuAuIAbY5/cRw4F1qvq3IvIV4GHAA1wAvqyqxzurz0LFmMjztrRw/qWXca9ZQ/OJE0QPHcrQpUtJWb6MmMzI/wKNhMbWRo7UHKGwspDCqkIOVR9qvygze0h2e8DkZuaSk5QT9h5eREJFRKKBYuBzQAWwH1jm/4teRP4GmKCqXxGRh4C/UNWlIuICCoFVqnpIRIYBdarqubZhwNdVda+IJKnqed/zC4C/UdW8zmq0UDGm91BVmg4cwL1mDQ1/3AXR0STNnk3q6nziJ0yIdHkR1eptpdhd3D5cVlhViPuSG4CUQSlXncY8btg4YqJiQlpPpEJlGvADVZ3te/w4gKr+s982O3zbvOkLko+AdGAOsFxVV3by+WOAV4CRek0jRGQZkK+qczqr0ULFmN6p5cMPqV23nrrnn8d74QLxkyaRujqfxM99rt+dktwdqkrZ+bL262UKKwupuFAB+C7KTJvQPi8zMX1i0C/KjFSoLALyVPUvfY9XAVNV9RG/bY76tqnwPT4FTAVW4gyJZeCEzEZV/ZdrPv97QJKqPur33MPAN4BY4LOq+n5nNVqoGNO7eS5cpP6FF3CvXUvr6dO4srJIWb6MlMWLiR7ae26v2xtUNVa1n2FWWFXISfdJFCVaohmfOp7JmZOZkjGFyZmTSY3r2erJkQqVxcDsa0LlblX9qt82x3zb+IfK3cD/wpkfuQtoBHYB31HVXX7vPY4zPHawg30v933u6g5e+zLwZYCRI0dOKS8vD1KLjTGhoh4PF/bsxb1mDY1vvYXExZG88AukrlrFoFtvjXR5vVJDSwOHqg+1z8scqT5Ci7cFgFFJo/jq5K8ya9Ssbn12Z6ESyn5kBTDC7/Fw4GyAbSp8w1/JgNv3/B5VrQEQkZeBXJxwQUQmAq6OAsVnI/DLjl5Q1aeAp8Dpqdx4s4wx4SbR0SR+diaJn53JpZMnca9dS/1vX6Bu4yYG33svqfn5DJ7+mQFxSnJXJcYmMj17OtOzpwPOHU2PnzvePi8zJGZISPYbyp6KC2ei/n7gDM5E/XJVPea3zcPAnX4T9Q+o6hIRScEJkOlAC7AdeFJVX/K978dAs6p+3++zxlwe7hKRzwPfD5Skl9nwlzF9V5vbTd2mTbifew5PdQ2xo0eTmu87JTlhYCzsGCmRPKV4LvBTnFOKn1HVfxSRHwEHVHWriMQBa4HJOD2Uh1S1xPfelcDjgAIvq+q3/D63BJirqu/5Pfcz4M+BVqAWeMQ/wDpioWJM36ctLZzfsQP3r5/l0rFjRCUnk7JkMSnLlxOTlRXp8volu/gxAAsVY/oPVaWpqAj3s2to2LkTREic9TlS8/OJnzTJhsaCKFJzKsYYEzYiQkJuLgm5ubSeOYN7/XPUbd5Mw7btxE2YQGp+PkmzZyExob2GY6Cznor1VIzpt7wXL1K3ZQu1a9bSUlaGKyODlOXLGbp0Ca6U/nmvk3Cw4a8ALFSMGRjU6+Xivn24n13DxTfeQAYNInnBAlLzVzFozJhIl9fnWKgEYKFizMDT/P77uNeuo37LFrS5mcGfnkbC1HtwpaXhSk/DlZZGdFoartRUu3o/AAuVACxUjBm42mprqfvNZmo3baTt7J8+voEI0ampTthc/kr3BU5a+lUhFJWUNKBOBLBQCcBCxRgD4L10ibaac3hqqmmrqXG+qmuufF9Tjcf3WFtaPvZ+iYkhOt0vbNLScKUN8wWQ7/nLARQfH4EWBped/WWMMZ2Iiosjdng2DM/udDtVxdvQ4Bc61XiuCaHWs2dpOnwYz7lz0MEf7VGDBztDbNeGkP/QW1o6rmF9c/it71VsjDERIiJEJyURnZTEoNGjO91W29rw1NZ20PPxBVF1Dc0nT3Lx9dfxNjR0tDOiU1KuO/TmSksjKjm51wy/WagYY0wIiMuFKz0dV3r6dbftyvBbS1lZwOE3YmKuhM+wYX4BdPXQmystLeRL2FioGGNMhAV1+O2jj2g6ejTw8FtCAtHpaaR/7Wskz5sX9LZYqBhjTB8RzOG3UF38aaFijDH90I0MvwVTVFj3Zowxpl+zUDHGGBM0FirGGGOCxkLFGGNM0FioGGOMCRoLFWOMMUFjoWKMMSZoLFSMMcYEzYBe+l5EqoHybr49DagJYjl9gbV5YLA2Dww9afMtqtrhVZUDOlR6QkQOBLqfQH9lbR4YrM0DQ6jabMNfxhhjgsZCxRhjTNBYqHTfU5EuIAKszQODtXlgCEmbbU7FGGNM0FhPxRhjTNBYqBhjjAkaC5XrEJE8ETkpIh+IyN918PpXROSIiLwrIq+JyCciUWcwXa/NftstEhEVkT5/KmYXjvMXRaTad5zfFZG/jESdwdSV4ywiS0TkuIgcE5Hnwl1jMHXhGD/pd3yLRaQuEnUGUxfaPFJEdotIkYgcFpG5Pd6pqtpXgC8gGjgFjAZigUPAJ67ZJsnv+wXA9kjXHeo2+7ZLBPYCbwGfinTdYTjOXwT+f6RrDXObxwBFQIrvcUak6w5le6/Z/qvAM5GuOwzH+Cngr33ffwIo6+l+rafSubuBD1S1RFVbgI3AF/w3UNXzfg8HA339zIfrttnnH4B/AS6Fs7gQ6Wqb+5OutPmvgJ+rai2AqlaFucZgutFjvAzYEJbKQqcrbVYgyfd9MnC2pzu1UOlcNvCh3+MK33NXEZGHReQUzi/Zr4WptlC5bptFZDIwQlVfDGdhIdSl4ww86BsiKBCREeEpLWS60uaxwFgReV1E3hKRvLBVF3xdPcaIyC1ADvBKGOoKpa60+QfAShGpAF7G6aH1iIVK56SD5z7WE1HVn6vqrcC3ge+EvKrQ6rTNIhIFPAl8M2wVhV5XjvPvgVGqOgH4I/BsyKsKra602YUzBHYfzl/u/yUiQ0NcV6h06WfZ5yGgQFU9IawnHLrS5mXAr1V1ODAXWOv7Ge82C5XOVQD+f5EOp/Pu4UZgYUgrCr3rtTkRuAN4VUTKgHuArX18sv66x1lVz6lqs+/h08CUMNUWKl35v10BbFHVVlUtBU7ihExfdCM/yw/R94e+oGtt/hLwGwBVfROIw1lostssVDq3HxgjIjkiEovzn22r/wYi4v9DNg94P4z1hUKnbVbVelVNU9VRqjoKZ6J+gaoeiEy5QdGV45zl93ABcCKM9YXCddsM/A6YCSAiaTjDYSVhrTJ4utJeROR2IAV4M8z1hUJX2nwauB9ARMbjhEp1T3bq6smb+ztVbRORR4AdOGdSPKOqx0TkR8ABVd0KPCIifw60ArXA6shV3HNdbHO/0sU2f01EFgBtgBvnbLA+q4tt3gHMEpHjgAd4TFXPRa7q7ruB/9fLgI3qOx2qL+tim78JPC0iX8cZGvtiT9tuy7QYY4wJGhv+MsYYEzQWKsYYY4LGQsUYY0zQWKgYY4wJGgsVY4wxQWOhYkyIiMgPROTRXlBHme86E2NCzkLFGGNM0FioGHMDRGSwiLwkIodE5KiILPXvCYjIp0TkVb+3TBSRV0TkfRH5K982WSKy13ffjqMicq/v+V+KyAHfvUt+6LfPMhH5JxF50/d6rojsEJFTIvIV3zb3+T7zBd/9T/6jozWcRGSliLzj2/d/ikh0KP+9zMBjoWLMjckDzqrqRFW9A9h+ne0n4CzfMw34nojcDCwHdqjqJGAi8K5v279X1U/53jNDRCb4fc6HqjoN2Af8GliEs+7aj/y2uRvnCuk7gVuBB/wL8S3DsRT4jG/fHmDFDbTdmOuyZVqMuTFHgH8Vkf8LvKiq+0Q6Wgy23RZVbQKaRGQ3zi/+/cAzIhID/E5VL4fKEhH5Ms7PZRbOTZMO+167vIzIEWCIqjYADSJyyW/l4HdUtQRARDYA04ECv1rux1kIc7+v5nigL98jxfRCFirG3ABVLRaRKTjLhP+ziPwBZz2wy73+uGvf8vGP0L0i8mc4PZi1IvITnB7Io8BdqlorIr++5rMur5Ds9fv+8uPLP8cf29c1jwV4VlUfv04zjek2G/4y5gb4hq8aVXUd8K9ALlDGlaXwH7zmLV8QkTgRGYZzX5L9vptAVanq08CvfJ+RBFwE6kUkE5jTjfLu9q1IG4UzzPXaNa/vAhaJSIavLam+WowJGuupGHNj7gR+IiJenJWp/xpnGOlXIvIE8PY1278DvASMBP5BVc+KyGrgMRFpBS4A+apaKiJFwDGc5eVf70ZtbwI/9tW4F3jB/0VVPS4i3wH+4AueVuBhoLwb+zKmQ7ZKsTH9gIjcBzyqqvMjXYsZ2Gz4yxhjTNBYT8UYY0zQWE/FGGNM0FioGGOMCRoLFWOMMUFjoWKMMSZoLFSMMcYEzf8ATZtActkBV8UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch3.best_score_, gsearch3.best_params_))\n",
    "test_means = gsearch3.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch3.cv_results_[ 'std_test_score' ]\n",
    "#train_means = gsearch3.cv_results_[ 'mean_train_score' ]\n",
    "#train_stds = gsearch3.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch3.cv_results_).to_csv('my_preds_subsampleh_colsample_bytree_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(colsample_bytree), len(subsample))\n",
    "#train_scores = np.array(train_means).reshape(len(colsample_bytree), len(subsample))\n",
    "\n",
    "for i, value in enumerate(colsample_bytree):\n",
    "    pyplot.plot(subsample, -test_scores[i], label= 'test_colsample_bytree:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'subsample' )                                                                                                      \n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'subsample_vs_colsample_bytree1.png' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\miniconda_anzhuang\\lib\\site-packages\\sklearn\\preprocessing\\label.py:219: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "d:\\miniconda_anzhuang\\lib\\site-packages\\sklearn\\preprocessing\\label.py:252: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "({'reg_alpha': 1.5, 'reg_lambda': 2}, -0.06759937675863166)"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#找出最佳 alpha & lambda\n",
    "\n",
    "reg_alpha = [ 1.5, 2]    #default = 0, 测试0.1,1，1.5，2\n",
    "reg_lambda = [0.5, 1, 2]      #default = 1，测试0.1， 0.5， 1，2\n",
    "\n",
    "param_test5_1 = dict(reg_alpha=reg_alpha, reg_lambda=reg_lambda)\n",
    "\n",
    "\n",
    "xgb5 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=98,  #第一轮参数调整得到的n_estimators最优值  98\n",
    "        max_depth=3,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.8,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n",
    "gsearch5_1 = GridSearchCV(xgb5, param_grid = param_test5_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch5_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch5_1.best_params_,     gsearch5_1.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#以上为参数调试过程，可以根据实际情况通过调整补长以及区间来不断的优化性能， 因为时间关系就不再进一步处理。\n",
    "\n",
    "最终模型参数如下：\n",
    "#reg_alpha': 1.5, 'reg_lambda': 1\n",
    "#colsample_bytree': 0.8, 'subsample': 0.8\n",
    "#learning_rate =0.1\n",
    "#n_estimators=98\n",
    "#max_depth=3\n",
    "#min_child_weight=1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=0.8,\n",
       "              colsample_bynode=1, colsample_bytree=0.8, gamma=0,\n",
       "              learning_rate=0.1, max_delta_step=0, max_depth=3,\n",
       "              min_child_weight=1, missing=None, n_estimators=98, n_jobs=1,\n",
       "              nthread=None, objective='binary:logistic', random_state=0,\n",
       "              reg_alpha=1.5, reg_lambda=0.8, scale_pos_weight=1, seed=3,\n",
       "              silent=None, subsample=0.3, verbosity=1)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb5 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=98,  #第一轮参数调整得到的n_estimators最优值  98\n",
    "        max_depth=3,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.8,\n",
    "        objective= 'binary:logistic',\n",
    "        reg_alpha=1.5,\n",
    "        reg_lambda=0.8,\n",
    "        seed=3)\n",
    "\n",
    "xgb5.fit(X_train,y_train,eval_metric='auc')\n",
    "\n",
    "#设置验证集合 verbose=False不打印过程\n",
    "xgb5.fit(X_train, y_train,eval_set=[(X_train, y_train)],eval_metric='auc',verbose=False)\n",
    "\n",
    "#获取验证集合结果\n",
    "#evals_result = xgb5.evals_result()\n",
    "#y_true, y_pred = y_test, xgb5.predict(X_test)\n",
    "#print\"Accuracy : %.4g\" % metrics.accuracy_score(y_true, y_pred)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#最终模型的特征重要性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>columns</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>Var5</td>\n",
       "      <td>0.124880</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Monthly_Income</td>\n",
       "      <td>0.124840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>Mobile_Verified</td>\n",
       "      <td>0.086767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>Device_Type</td>\n",
       "      <td>0.081432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>Filled_Form</td>\n",
       "      <td>0.070331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>Source</td>\n",
       "      <td>0.066999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>Existing_EMI</td>\n",
       "      <td>0.059087</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>Var4</td>\n",
       "      <td>0.055614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>Loan_Tenure_Applied</td>\n",
       "      <td>0.048212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Gender</td>\n",
       "      <td>0.047051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>Interest_Rate</td>\n",
       "      <td>0.045959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Loan_Amount_Applied</td>\n",
       "      <td>0.043130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>EMI_Loan_Submitted</td>\n",
       "      <td>0.036002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>Loan_Amount_Submitted</td>\n",
       "      <td>0.035381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>Processing_Fee</td>\n",
       "      <td>0.033265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>Loan_Tenure_Submitted</td>\n",
       "      <td>0.027778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>Var1</td>\n",
       "      <td>0.011970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>Salary_Account</td>\n",
       "      <td>0.001303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>City</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>Employer_Name</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>Var2</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  columns  importance\n",
       "9                    Var5    0.124880\n",
       "2          Monthly_Income    0.124840\n",
       "8         Mobile_Verified    0.086767\n",
       "17            Device_Type    0.081432\n",
       "16            Filled_Form    0.070331\n",
       "19                 Source    0.066999\n",
       "5            Existing_EMI    0.059087\n",
       "20                   Var4    0.055614\n",
       "4     Loan_Tenure_Applied    0.048212\n",
       "0                  Gender    0.047051\n",
       "13          Interest_Rate    0.045959\n",
       "3     Loan_Amount_Applied    0.043130\n",
       "15     EMI_Loan_Submitted    0.036002\n",
       "11  Loan_Amount_Submitted    0.035381\n",
       "14         Processing_Fee    0.033265\n",
       "12  Loan_Tenure_Submitted    0.027778\n",
       "10                   Var1    0.011970\n",
       "7          Salary_Account    0.001303\n",
       "1                    City    0.000000\n",
       "6           Employer_Name    0.000000\n",
       "18                   Var2    0.000000"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({\"columns\":list(X_train.columns), \"importance\":list(xgb5.feature_importances_)})\n",
    "df.sort_values(by=['importance'],ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
